alpha-id/docs/superpowers/plans/2026-05-18-alpha-id-icd-suggestion.md

alpha-id ICD Code Suggestion — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: A Rust core library + thin CLI that turns a list of caller-extracted diagnostic phrases (one per line) into one prioritized list of ≤10 ICD-10-GM codes, with tag filtering, validated by a noise+paraphrase eval harness.

Architecture: Two modes behind one interface. Mode C = lexical baseline (tantivy BM25 over the ~90k Alpha-ID phrase corpus) + ClaML tag filter. Mode A = adds IONOS bge-m3 semantic retrieval (brute-force cosine over precomputed embeddings) fused with lexical via Reciprocal Rank Fusion, then IONOS Qwen3-VL-Reranker-8B cross-encoder rerank. Cross-unit fusion (one unit = one extracted phrase) produces one global top-10. Embedding is one-time, hash-cached, resumable, cost-guarded. On IONOS failure mode A degrades to C.

Tech Stack: Rust 1.94, clap (CLI), tantivy (lexical), quick-xml (ClaML streaming parse), reqwest blocking (IONOS HTTP), serde/serde_json/toml, sha2 (cache keys), rand (seeded error injection). No ANN crate — brute-force exact cosine over 90k×1024 f32 is <50 ms and removes a native-build dependency (YAGNI).

Spec: docs/superpowers/specs/2026-05-18-alpha-id-icd-suggestion-design.md. Data lives in gitignored data/ and icd-claml/ (already present).

Revision 2026-05-18 (Input-Vertrag), spec §1a: The input is no longer a blank-line-segmented dictation but a list of caller-extracted diagnostic phrases, one per non-empty line (extraction is doctate's LLM, never this project). Evidence-driven: terse corpus strings need terse query input; whole-paragraph embedding dilutes the diagnosis (see spec §1a). Scope of code change is deliberately minimal: only Task 6 (segment.rs) changes — split on newlines instead of blank lines; function name split_segments and the segment_idx/source_segments fields are retained (a "segment" now denotes one extracted phrase). The pipeline core (Tasks 9/10/18) is unchanged. Eval Tasks 11 & 19 gain paraphrase injection + a hand-labeled phrase-list set + separated reporting of extraction-recall ceiling vs. matcher recall. Where this plan still says "dictation/blank lines/Befund segment", read "phrase list / newlines / extracted-phrase unit".

---

File Structure

Cargo.toml                       # single binary+lib crate
config/default.toml              # IONOS endpoint, model names, pool sizes, paths
src/lib.rs                       # library root, module decls, re-exports
src/model.rs                     # core types: AlphaIdEntry, IcdMeta, Tags, Candidate, Suggestion, Filter, Mode, Config, Diagnostics, AppError
src/corpus.rs                    # parse Alpha-ID-SE -> Vec<AlphaIdEntry> + helpers
src/claml.rs                     # stream-parse ClaML XML -> HashMap<String, IcdMeta>
src/normalize.rs                 # text normalization + abbreviation expansion
src/segment.rs                   # split phrase list: one non-empty line -> one unit (rev 2026-05-18)
src/lexical.rs                   # tantivy index build/load/query over Alpha-ID texts
src/tags.rs                      # build Tags from IcdMeta+validity; Filter predicate
src/fusion.rs                    # RRF within segment; cross-segment dedupe(max)
src/ionos.rs                     # shared IONOS HTTP client: token, retry/backoff
src/embed.rs                     # bge-m3 client + sha256-keyed resumable disk cache + cost estimate
src/vector.rs                    # load embeddings; brute-force cosine top-k
src/rerank.rs                    # Qwen3-VL-Reranker-8B pair scoring client
src/pipeline.rs                  # orchestrate [1]-[7] for Mode C and Mode A (+ degrade)
src/eval.rs                      # error injector + gold-set generator + Recall@k
src/bin/alpha_id.rs              # CLI: index | suggest | eval
tests/corpus_tests.rs            # integration: real data/ file
tests/claml_tests.rs            # integration: real icd-claml/ file
tests/pipeline_mode_c_tests.rs   # integration: end-to-end Mode C

Each file = one responsibility. The library is CLI-agnostic so doctate can later embed it unchanged. IONOS-touching tests use a local mock HTTP server (tiny_http dev-dependency), never the real endpoint.

---

Phase 0 — Scaffold

Task 1: Cargo project skeleton

Files:

• Create: Cargo.toml

• Create: src/lib.rs

• Create: src/bin/alpha_id.rs

• Create: config/default.toml

• Step 1: Create Cargo.toml

[package]
name = "alpha-id"
version = "0.1.0"
edition = "2021"

[lib]
name = "alpha_id"
path = "src/lib.rs"

[[bin]]
name = "alpha-id"
path = "src/bin/alpha_id.rs"

[dependencies]
clap = { version = "4", features = ["derive"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
toml = "0.8"
quick-xml = "0.36"
tantivy = "0.22"
reqwest = { version = "0.12", default-features = false, features = ["blocking", "json", "rustls-tls"] }
sha2 = "0.10"
rand = "0.8"
thiserror = "1"

[dev-dependencies]
tiny_http = "0.12"
tempfile = "3"

• Step 2: Create src/lib.rs

pub mod model;
pub mod corpus;
pub mod claml;
pub mod normalize;
pub mod segment;
pub mod lexical;
pub mod tags;
pub mod fusion;
pub mod ionos;
pub mod embed;
pub mod vector;
pub mod rerank;
pub mod pipeline;
pub mod eval;

• Step 3: Create config/default.toml

ionos_base_url = "https://openai.inference.de-txl.ionos.com/v1"
token_path = "~/.ionos_token"
embed_model = "BAAI/bge-m3"
rerank_model = "Qwen/Qwen3-VL-Reranker-8B"
alpha_id_path = "data/icd10gm2026_alphaidse_edvtxt_20250926.txt"
claml_path = "icd-claml/Klassifikationsdateien/icd10gm2026syst_claml_20250912.xml"
index_dir = "index"
pool_size = 60
top_k = 10

• Step 4: Create placeholder src/bin/alpha_id.rs

fn main() {
    println!("alpha-id CLI — see `alpha-id --help`");
}

• Step 5: Stub the remaining modules so lib.rs compiles

Create each of src/model.rs, src/corpus.rs, src/claml.rs, src/normalize.rs, src/segment.rs, src/lexical.rs, src/tags.rs, src/fusion.rs, src/ionos.rs, src/embed.rs, src/vector.rs, src/rerank.rs, src/pipeline.rs, src/eval.rs containing exactly one line:

// implemented in a later task

• Step 6: Verify it builds

Run: cargo build Expected: compiles with warnings about empty modules, no errors.

• Step 7: Commit

git add Cargo.toml Cargo.lock src/ config/
git commit -m "chore: scaffold alpha-id crate skeleton"

---

Phase C — Lexical baseline + data layer + eval (fully working, no IONOS)

Task 2: Core types

Files:

• Modify: src/model.rs

• Test: tests/model_tests.rs

• Step 1: Write the failing test — tests/model_tests.rs

use alpha_id::model::{Filter, Tags, Mode};

#[test]
fn filter_billable_only_rejects_non_primary() {
    let tags = Tags { billable: false, valid: true, chapter: "I".into(), exotic: false };
    let f = Filter { billable_only: true, valid_only: false, chapters: None, exclude_exotic: false };
    assert!(!f.accepts(&tags));
}

#[test]
fn filter_default_accepts_everything() {
    let tags = Tags { billable: false, valid: false, chapter: "II".into(), exotic: true };
    assert!(Filter::default().accepts(&tags));
}

#[test]
fn mode_parses_from_str() {
    assert_eq!("a".parse::<Mode>().unwrap(), Mode::A);
    assert_eq!("c".parse::<Mode>().unwrap(), Mode::C);
}

• Step 2: Run test to verify it fails

Run: cargo test --test model_tests Expected: FAIL — model items not found.

• Step 3: Write minimal implementation — src/model.rs

use serde::{Deserialize, Serialize};
use std::str::FromStr;

#[derive(Debug, Clone)]
pub struct AlphaIdEntry {
    pub alpha_id: String,
    pub valid: bool,
    pub icd_primary: String,   // Field 3, normalized below
    pub icd_star: String,      // Field 4
    pub icd_addon: String,     // Field 5 ("!") — NOT ICD exclusion
    pub icd_primary2: String,  // Field 6
    pub orpha: String,         // Field 7
    pub text: String,          // Field 8
}

#[derive(Debug, Clone, Default)]
pub struct IcdMeta {
    pub code: String,          // normalized, no + * !
    pub description: String,
    pub chapter: String,
    pub group: String,
    pub para295: String,       // P/O/Z/V
    pub para301: String,
    pub exotic: bool,
    pub ifsg: bool,
    pub content: bool,
}

#[derive(Debug, Clone, Serialize)]
pub struct Tags {
    pub billable: bool,        // para295 == "P"
    pub valid: bool,           // Alpha-ID field 1
    pub chapter: String,
    pub exotic: bool,
}

#[derive(Debug, Clone)]
pub struct Filter {
    pub billable_only: bool,
    pub valid_only: bool,
    pub chapters: Option<Vec<String>>,
    pub exclude_exotic: bool,
}

impl Default for Filter {
    fn default() -> Self {
        Filter { billable_only: false, valid_only: false, chapters: None, exclude_exotic: false }
    }
}

impl Filter {
    pub fn accepts(&self, t: &Tags) -> bool {
        if self.billable_only && !t.billable { return false; }
        if self.valid_only && !t.valid { return false; }
        if self.exclude_exotic && t.exotic { return false; }
        if let Some(ch) = &self.chapters {
            if !ch.iter().any(|c| c == &t.chapter) { return false; }
        }
        true
    }
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Mode { C, A }

impl FromStr for Mode {
    type Err = String;
    fn from_str(s: &str) -> Result<Self, String> {
        match s.to_ascii_lowercase().as_str() {
            "c" => Ok(Mode::C),
            "a" => Ok(Mode::A),
            other => Err(format!("unknown mode: {other}")),
        }
    }
}

#[derive(Debug, Clone)]
pub struct Candidate {
    pub icd_code: String,
    pub alpha_text: String,
    pub segment_idx: usize,
    pub lexical: Option<f32>,
    pub semantic: Option<f32>,
    pub rerank: Option<f32>,
}

#[derive(Debug, Clone, Serialize)]
pub struct Suggestion {
    pub icd_code: String,
    pub description: String,
    pub score: f32,
    pub matched_phrase: String,
    pub source_segments: Vec<usize>,
    pub tags: Tags,
}

#[derive(Debug, Clone, Serialize, Default)]
pub struct Diagnostics {
    pub mode: String,
    pub degraded: bool,
    pub segment_count: usize,
    pub ionos_calls: usize,
    pub millis: u128,
}

#[derive(Debug, Serialize)]
pub struct SuggestResult {
    pub suggestions: Vec<Suggestion>,
    pub diagnostics: Diagnostics,
}

#[derive(Debug, Clone, Deserialize)]
pub struct Config {
    pub ionos_base_url: String,
    pub token_path: String,
    pub embed_model: String,
    pub rerank_model: String,
    pub alpha_id_path: String,
    pub claml_path: String,
    pub index_dir: String,
    pub pool_size: usize,
    pub top_k: usize,
}

impl Config {
    pub fn load(path: &str) -> Result<Config, AppError> {
        let s = std::fs::read_to_string(path)
            .map_err(|e| AppError::Io(format!("config {path}: {e}")))?;
        toml::from_str(&s).map_err(|e| AppError::Parse(format!("config: {e}")))
    }
}

#[derive(Debug, thiserror::Error)]
pub enum AppError {
    #[error("io error: {0}")]
    Io(String),
    #[error("parse error: {0}")]
    Parse(String),
    #[error("ionos error: {0}")]
    Ionos(String),
    #[error("config error: {0}")]
    Config(String),
}

• Step 4: Run test to verify it passes

Run: cargo test --test model_tests Expected: PASS (3 tests).

• Step 5: Commit

git add src/model.rs tests/model_tests.rs
git commit -m "feat: core domain types and Filter predicate"

---

Task 3: Alpha-ID corpus parser

Files:

• Modify: src/corpus.rs

• Test: tests/corpus_tests.rs

• Step 1: Write the failing test — tests/corpus_tests.rs

use alpha_id::corpus;

#[test]
fn parses_pipe_line_into_entry() {
    let line = "1|I69812|A01.0+|I39.8*|||99745|Typhus-Endokarditis";
    let e = corpus::parse_line(line).expect("should parse");
    assert_eq!(e.alpha_id, "I69812");
    assert!(e.valid);
    assert_eq!(e.icd_primary, "A01.0"); // normalized: no '+'
    assert_eq!(e.icd_star, "I39.8");    // normalized: no '*'
    assert_eq!(e.orpha, "99745");
    assert_eq!(e.text, "Typhus-Endokarditis");
}

#[test]
fn invalid_flag_is_false() {
    let line = "0|I97837|||U81!|||Bakterien mit Multiresistenz";
    let e = corpus::parse_line(line).unwrap();
    assert!(!e.valid);
    assert_eq!(e.icd_addon, "U81"); // field 5, '!' stripped
    assert_eq!(e.icd_primary, "");
}

#[test]
fn loads_full_real_corpus() {
    let entries = corpus::load("data/icd10gm2026_alphaidse_edvtxt_20250926.txt").unwrap();
    assert_eq!(entries.len(), 90_898);
    assert!(entries.iter().any(|e| e.text.contains("Diabetes mellitus Typ 2")));
}

• Step 2: Run test to verify it fails

Run: cargo test --test corpus_tests Expected: FAIL — corpus::parse_line not found.

• Step 3: Write minimal implementation — src/corpus.rs

use crate::model::{AlphaIdEntry, AppError};

/// Normalize an ICD code field by removing classification marks.
pub fn normalize_code(raw: &str) -> String {
    raw.trim().trim_end_matches(['+', '*', '!']).trim().to_string()
}

pub fn parse_line(line: &str) -> Option<AlphaIdEntry> {
    let f: Vec<&str> = line.split('|').collect();
    if f.len() < 8 { return None; }
    Some(AlphaIdEntry {
        valid: f[0].trim() == "1",
        alpha_id: f[1].trim().to_string(),
        icd_primary: normalize_code(f[2]),
        icd_star: normalize_code(f[3]),
        icd_addon: normalize_code(f[4]),
        icd_primary2: normalize_code(f[5]),
        orpha: f[6].trim().to_string(),
        text: f[7].trim().to_string(),
    })
}

pub fn load(path: &str) -> Result<Vec<AlphaIdEntry>, AppError> {
    let raw = std::fs::read_to_string(path)
        .map_err(|e| AppError::Io(format!("{path}: {e}")))?;
    let entries: Vec<AlphaIdEntry> = raw
        .lines()
        .filter(|l| !l.trim().is_empty())
        .filter_map(parse_line)
        .collect();
    if entries.is_empty() {
        return Err(AppError::Parse(format!("no entries parsed from {path}")));
    }
    Ok(entries)
}

/// The primary ICD code a matched entry contributes (falls back to star/addon).
pub fn primary_code(e: &AlphaIdEntry) -> Option<&str> {
    for c in [&e.icd_primary, &e.icd_primary2, &e.icd_star, &e.icd_addon] {
        if !c.is_empty() { return Some(c); }
    }
    None
}

• Step 4: Run test to verify it passes

Run: cargo test --test corpus_tests Expected: PASS (3 tests). The full-corpus test reads the real gitignored file.

• Step 5: Commit

git add src/corpus.rs tests/corpus_tests.rs
git commit -m "feat: Alpha-ID-SE corpus parser with code normalization"

---

Task 4: ClaML metadata parser

Files:

• Modify: src/claml.rs

• Test: tests/claml_tests.rs

• Step 1: Write the failing test — tests/claml_tests.rs

use alpha_id::claml;

#[test]
fn parses_real_claml_meta_and_titles() {
    let map = claml::load("icd-claml/Klassifikationsdateien/icd10gm2026syst_claml_20250912.xml").unwrap();
    let a010 = map.get("A01.0").expect("A01.0 present");
    assert_eq!(a010.para295, "P");          // billable primary
    assert_eq!(a010.chapter, "01");
    assert!(a010.description.contains("Typhus abdominalis"));
    let a01 = map.get("A01").expect("A01 present");
    assert_eq!(a01.para295, "V");           // 3-digit non-codable
    assert!(map.len() > 10_000);
}

• Step 2: Run test to verify it fails

Run: cargo test --test claml_tests Expected: FAIL — claml::load not found.

• Step 3: Write minimal implementation — src/claml.rs

use crate::model::{AppError, IcdMeta};
use quick_xml::events::Event;
use quick_xml::reader::Reader;
use std::collections::HashMap;

/// Stream-parse the ClaML XML. We only need <Class kind="category">:
/// its code attr, <Meta name/value> children, owning chapter, and the
/// preferred <Rubric><Label> text.
pub fn load(path: &str) -> Result<HashMap<String, IcdMeta>, AppError> {
    let mut reader = Reader::from_file(path)
        .map_err(|e| AppError::Io(format!("{path}: {e}")))?;
    reader.config_mut().trim_text(true);

    let mut map: HashMap<String, IcdMeta> = HashMap::new();
    let mut buf = Vec::new();

    let mut cur: Option<IcdMeta> = None;
    let mut cur_kind = String::new();
    let mut in_preferred = false;
    let mut rubric_kind = String::new();

    loop {
        match reader.read_event_into(&mut buf) {
            Err(e) => return Err(AppError::Parse(format!("claml: {e}"))),
            Ok(Event::Eof) => break,
            Ok(Event::Start(t)) | Ok(Event::Empty(t)) => {
                let name = t.name();
                match name.as_ref() {
                    b"Class" => {
                        let mut code = String::new();
                        cur_kind.clear();
                        for a in t.attributes().flatten() {
                            match a.key.as_ref() {
                                b"code" => code = String::from_utf8_lossy(&a.value).into_owned(),
                                b"kind" => cur_kind = String::from_utf8_lossy(&a.value).into_owned(),
                                _ => {}
                            }
                        }
                        if cur_kind == "category" {
                            let mut m = IcdMeta::default();
                            m.code = code;
                            cur = Some(m);
                        } else {
                            cur = None;
                        }
                    }
                    b"SuperClass" => {
                        if let Some(m) = cur.as_mut() {
                            for a in t.attributes().flatten() {
                                if a.key.as_ref() == b"code" {
                                    let sup = String::from_utf8_lossy(&a.value);
                                    if m.group.is_empty() { m.group = sup.into_owned(); }
                                }
                            }
                        }
                    }
                    b"Meta" => {
                        if let Some(m) = cur.as_mut() {
                            let mut k = String::new();
                            let mut v = String::new();
                            for a in t.attributes().flatten() {
                                match a.key.as_ref() {
                                    b"name" => k = String::from_utf8_lossy(&a.value).into_owned(),
                                    b"value" => v = String::from_utf8_lossy(&a.value).into_owned(),
                                    _ => {}
                                }
                            }
                            match k.as_str() {
                                "Para295" => m.para295 = v,
                                "Para301" => m.para301 = v,
                                "Exotic" => m.exotic = v == "J",
                                "Infectious" => m.ifsg = v == "J",
                                "Content" => m.content = v == "J",
                                "chapter" => m.chapter = v,
                                _ => {}
                            }
                        }
                    }
                    b"Rubric" => {
                        rubric_kind.clear();
                        for a in t.attributes().flatten() {
                            if a.key.as_ref() == b"kind" {
                                rubric_kind = String::from_utf8_lossy(&a.value).into_owned();
                            }
                        }
                        in_preferred = rubric_kind == "preferred";
                    }
                    _ => {}
                }
            }
            Ok(Event::Text(txt)) => {
                if in_preferred {
                    if let Some(m) = cur.as_mut() {
                        if m.description.is_empty() {
                            m.description = txt.unescape().unwrap_or_default().into_owned();
                        }
                    }
                }
            }
            Ok(Event::End(t)) => {
                match t.name().as_ref() {
                    b"Rubric" => in_preferred = false,
                    b"Class" => {
                        if let Some(m) = cur.take() {
                            if !m.code.is_empty() { map.insert(m.code.clone(), m); }
                        }
                    }
                    _ => {}
                }
            }
            _ => {}
        }
        buf.clear();
    }
    if map.is_empty() {
        return Err(AppError::Parse(format!("no ClaML categories in {path}")));
    }
    Ok(map)
}

Note: chapter may arrive via a chapter Meta or be derivable from the owning <Class kind="chapter">. If the real file lacks a chapter Meta on categories, derive it: keep a stack of the current chapter code while walking; assign m.chapter from the nearest enclosing kind="chapter" Class code. Implement the stack only if Step 4 shows empty chapters.

• Step 2b (conditional): chapter-stack fallback

If cargo test --test claml_tests shows a010.chapter empty, add a chapter_stack: Vec<String> pushed on <Class kind="chapter"> start (store its code), popped on its End, and set m.chapter = chapter_stack.last().cloned().unwrap_or_default() when creating a category. Re-run.

• Step 3: Run test to verify it passes

Run: cargo test --test claml_tests Expected: PASS. If chapter assertion fails, apply Step 2b then re-run.

• Step 4: Commit

git add src/claml.rs tests/claml_tests.rs
git commit -m "feat: streaming ClaML parser for ICD metadata and titles"

---

Task 5: Text normalization

Files:

• Modify: src/normalize.rs

• Test: tests/normalize_tests.rs

• Step 1: Write the failing test — tests/normalize_tests.rs

use alpha_id::normalize::normalize;

#[test]
fn lowercases_and_collapses_whitespace() {
    assert_eq!(normalize("  Diabetes   Mellitus\tTyp 2 "), "diabetes mellitus typ 2");
}

#[test]
fn expands_common_medical_abbreviations() {
    assert_eq!(normalize("V.a. art. Hypertonie"), "verdacht auf arterielle hypertonie");
    assert_eq!(normalize("Z.n. OP"), "zustand nach operation");
}

#[test]
fn keeps_german_umlauts() {
    assert_eq!(normalize("Ödem"), "ödem");
}

• Step 2: Run test to verify it fails

Run: cargo test --test normalize_tests Expected: FAIL — normalize not found.

• Step 3: Write minimal implementation — src/normalize.rs

/// (pattern, replacement) — applied as whole-token, case-insensitive.
const ABBREV: &[(&str, &str)] = &[
    ("v.a.", "verdacht auf"),
    ("z.n.", "zustand nach"),
    ("art.", "arterielle"),
    ("op", "operation"),
    ("dd", "differentialdiagnose"),
    ("re.", "rechts"),
    ("li.", "links"),
];

pub fn normalize(input: &str) -> String {
    let lower = input.to_lowercase();
    let mut out: Vec<String> = Vec::new();
    for tok in lower.split_whitespace() {
        let replaced = ABBREV.iter()
            .find(|(p, _)| *p == tok)
            .map(|(_, r)| r.to_string())
            .unwrap_or_else(|| tok.to_string());
        out.push(replaced);
    }
    out.join(" ")
}

• Step 4: Run test to verify it passes

Run: cargo test --test normalize_tests Expected: PASS (3 tests).

• Step 5: Commit

git add src/normalize.rs tests/normalize_tests.rs
git commit -m "feat: text normalization with medical abbreviation expansion"

---

Task 6: Phrase-list splitting (rev 2026-05-18, spec §1a)

Input is a caller-extracted phrase list: one non-empty (trimmed) line = one retrieval unit. Blank/whitespace-only lines are skipped — they are no longer paragraph delimiters. Function name split_segments and the Vec<String> shape are retained (minimal-churn decision, spec §3 note); a "segment" now denotes one extracted phrase.

Files:

• Modify: src/segment.rs

• Test: tests/segment_tests.rs

• Step 1: Write the failing test — tests/segment_tests.rs

use alpha_id::segment::split_segments;

#[test]
fn one_unit_per_nonempty_line() {
    let input = "arterielle Hypertonie\nTyp-2-Diabetes mit Polyneuropathie\ngemischte Hyperlipidämie";
    assert_eq!(split_segments(input), vec![
        "arterielle Hypertonie".to_string(),
        "Typ-2-Diabetes mit Polyneuropathie".to_string(),
        "gemischte Hyperlipidämie".to_string(),
    ]);
}

#[test]
fn blank_and_whitespace_lines_are_skipped_and_each_line_trimmed() {
    let input = "  \n\n  Kreuzschmerz \n   \nGonarthrose rechts\n\n";
    assert_eq!(split_segments(input), vec![
        "Kreuzschmerz".to_string(),
        "Gonarthrose rechts".to_string(),
    ]);
}

#[test]
fn single_phrase_is_one_unit() {
    assert_eq!(split_segments("nur eine Diagnose"), vec!["nur eine Diagnose".to_string()]);
}

#[test]
fn empty_or_whitespace_only_input_yields_no_units() {
    assert!(split_segments("   \n\n").is_empty());
}

• Step 2: Run test to verify it fails

Run: cargo test --test segment_tests Expected: FAIL — old blank-line-paragraph behavior joins lines / new test names absent.

• Step 3: Write minimal implementation — src/segment.rs

/// Split the caller-extracted phrase list into retrieval units: one
/// non-empty (trimmed) input line = one unit. Blank/whitespace-only
/// lines are skipped. Rev 2026-05-18 (spec §1a): input is extracted
/// diagnostic phrases, not a blank-line-segmented dictation. Name and
/// signature kept; a "segment" now denotes one extracted phrase.
pub fn split_segments(phrases: &str) -> Vec<String> {
    phrases
        .lines()
        .map(|l| l.trim())
        .filter(|l| !l.is_empty())
        .map(|l| l.to_string())
        .collect()
}

• Step 4: Run test to verify it passes

Run: cargo test --test segment_tests Expected: PASS (4 tests).

• Step 5: Commit

git add src/segment.rs tests/segment_tests.rs
git commit -m "refactor: input contract — one extracted phrase per line (spec §1a)"

---

Task 7: Tags builder

Files:

• Modify: src/tags.rs

• Test: tests/tags_tests.rs

• Step 1: Write the failing test — tests/tags_tests.rs

use alpha_id::model::IcdMeta;
use alpha_id::tags::tags_for;

#[test]
fn billable_when_para295_is_p_and_valid() {
    let mut m = IcdMeta::default();
    m.para295 = "P".into();
    m.chapter = "01".into();
    let t = tags_for(&m, true);
    assert!(t.billable);
    assert!(t.valid);
    assert_eq!(t.chapter, "01");
}

#[test]
fn not_billable_when_para295_is_v() {
    let mut m = IcdMeta::default();
    m.para295 = "V".into();
    assert!(!tags_for(&m, true).billable);
}

• Step 2: Run test to verify it fails

Run: cargo test --test tags_tests Expected: FAIL — tags_for not found.

• Step 3: Write minimal implementation — src/tags.rs

use crate::model::{IcdMeta, Tags};

/// Build display/filter Tags from ICD metadata and the Alpha-ID
/// validity flag of the matched entry.
pub fn tags_for(m: &IcdMeta, alpha_valid: bool) -> Tags {
    Tags {
        billable: m.para295 == "P",
        valid: alpha_valid,
        chapter: m.chapter.clone(),
        exotic: m.exotic,
    }
}

• Step 4: Run test to verify it passes

Run: cargo test --test tags_tests Expected: PASS (2 tests).

• Step 5: Commit

git add src/tags.rs tests/tags_tests.rs
git commit -m "feat: tag derivation from ClaML metadata + validity"

---

Task 8: Lexical index (tantivy)

Files:

• Modify: src/lexical.rs

• Test: tests/lexical_tests.rs

• Step 1: Write the failing test — tests/lexical_tests.rs

use alpha_id::lexical::LexicalIndex;
use alpha_id::model::AlphaIdEntry;

fn entry(id: &str, code: &str, text: &str) -> AlphaIdEntry {
    AlphaIdEntry {
        alpha_id: id.into(), valid: true,
        icd_primary: code.into(), icd_star: "".into(), icd_addon: "".into(),
        icd_primary2: "".into(), orpha: "".into(), text: text.into(),
    }
}

#[test]
fn finds_entry_by_fuzzy_query() {
    let entries = vec![
        entry("I1", "E11.9", "Diabetes mellitus Typ 2"),
        entry("I2", "I10.90", "Arterielle essentielle Hypertonie"),
    ];
    let idx = LexicalIndex::build_in_ram(&entries).unwrap();
    let hits = idx.search("diabetes typ 2", 5).unwrap();
    assert!(!hits.is_empty());
    assert_eq!(hits[0].0, 0); // entry index 0
    assert!(hits[0].1 > 0.0); // bm25 score
}

• Step 2: Run test to verify it fails

Run: cargo test --test lexical_tests Expected: FAIL — LexicalIndex not found.

• Step 3: Write minimal implementation — src/lexical.rs

use crate::model::{AlphaIdEntry, AppError};
use crate::normalize::normalize;
use tantivy::collector::TopDocs;
use tantivy::query::QueryParser;
use tantivy::schema::{Schema, STORED, TEXT, Value};
use tantivy::{doc, Index, TantivyDocument};

pub struct LexicalIndex {
    index: Index,
    text_field: tantivy::schema::Field,
    idx_field: tantivy::schema::Field,
}

impl LexicalIndex {
    fn schema() -> (Schema, tantivy::schema::Field, tantivy::schema::Field) {
        let mut b = Schema::builder();
        let text_field = b.add_text_field("text", TEXT);
        let idx_field = b.add_u64_field("idx", STORED);
        (b.build(), text_field, idx_field)
    }

    pub fn build_in_ram(entries: &[AlphaIdEntry]) -> Result<Self, AppError> {
        let (schema, text_field, idx_field) = Self::schema();
        let index = Index::create_in_ram(schema);
        let mut w = index.writer(50_000_000)
            .map_err(|e| AppError::Io(format!("tantivy writer: {e}")))?;
        for (i, e) in entries.iter().enumerate() {
            w.add_document(doc!(text_field => normalize(&e.text), idx_field => i as u64))
                .map_err(|e| AppError::Io(format!("tantivy add: {e}")))?;
        }
        w.commit().map_err(|e| AppError::Io(format!("tantivy commit: {e}")))?;
        Ok(Self { index, text_field, idx_field })
    }

    /// Returns (entry_index, bm25_score) descending.
    pub fn search(&self, query: &str, k: usize) -> Result<Vec<(usize, f32)>, AppError> {
        let reader = self.index.reader()
            .map_err(|e| AppError::Io(format!("tantivy reader: {e}")))?;
        let searcher = reader.searcher();
        let mut qp = QueryParser::for_index(&self.index, vec![self.text_field]);
        qp.set_conjunction_by_default(); // all terms; falls back via OR below
        let norm = normalize(query);
        let parsed = QueryParser::for_index(&self.index, vec![self.text_field])
            .parse_query(&format!("{}", norm.replace(['(', ')', '"', ':'], " ")))
            .or_else(|_| qp.parse_query(&norm))
            .map_err(|e| AppError::Parse(format!("query: {e}")))?;
        let top = searcher.search(&parsed, &TopDocs::with_limit(k.max(1)))
            .map_err(|e| AppError::Io(format!("tantivy search: {e}")))?;
        let mut out = Vec::new();
        for (score, addr) in top {
            let d: TantivyDocument = searcher.doc(addr)
                .map_err(|e| AppError::Io(format!("tantivy doc: {e}")))?;
            if let Some(v) = d.get_first(self.idx_field).and_then(|v| v.as_u64()) {
                out.push((v as usize, score));
            }
        }
        Ok(out)
    }
}

• Step 4: Run test to verify it passes

Run: cargo test --test lexical_tests Expected: PASS. If the query parser rejects the input, the .or_else fallback handles it; adjust the sanitizing replace set if a panic shows another special char.

• Step 5: Commit

git add src/lexical.rs tests/lexical_tests.rs
git commit -m "feat: tantivy lexical index over Alpha-ID texts"

---

Task 9: Fusion (RRF + cross-segment)

Files:

• Modify: src/fusion.rs

• Test: tests/fusion_tests.rs

• Step 1: Write the failing test — tests/fusion_tests.rs

use alpha_id::fusion::{rrf_merge, cross_segment_dedupe};
use alpha_id::model::Candidate;

fn cand(code: &str, seg: usize, rerank: Option<f32>) -> Candidate {
    Candidate { icd_code: code.into(), alpha_text: code.into(),
        segment_idx: seg, lexical: None, semantic: None, rerank }
}

#[test]
fn rrf_merges_two_rankings_by_reciprocal_rank() {
    let lex = vec![0usize, 1, 2];   // entry indices ranked
    let sem = vec![2usize, 0, 3];
    let merged = rrf_merge(&lex, &sem, 60);
    // entry 0 appears rank0 in lex and rank1 in sem -> highest
    assert_eq!(merged[0], 0);
    assert!(merged.contains(&3));
}

#[test]
fn cross_segment_keeps_max_score_and_all_sources() {
    let cands = vec![
        cand("E11.9", 0, Some(0.4)),
        cand("E11.9", 2, Some(0.9)),
        cand("I10.90", 1, Some(0.5)),
    ];
    let mut merged = cross_segment_dedupe(cands);
    merged.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap());
    assert_eq!(merged[0].0.icd_code, "E11.9");
    assert_eq!(merged[0].1, 0.9);                       // max
    assert_eq!(merged[0].0.source_segments, vec![0, 2]); // all sources
}

• Step 2: Run test to verify it fails

Run: cargo test --test fusion_tests Expected: FAIL — fusion items not found.

• Step 3: Write minimal implementation — src/fusion.rs

use crate::model::Candidate;
use std::collections::HashMap;

/// Reciprocal Rank Fusion of two ranked lists of entry indices.
/// Returns deduped entry indices ordered by fused score desc.
pub fn rrf_merge(a: &[usize], b: &[usize], k: usize) -> Vec<usize> {
    let mut score: HashMap<usize, f64> = HashMap::new();
    for (rank, &id) in a.iter().enumerate() {
        *score.entry(id).or_default() += 1.0 / (k as f64 + rank as f64 + 1.0);
    }
    for (rank, &id) in b.iter().enumerate() {
        *score.entry(id).or_default() += 1.0 / (k as f64 + rank as f64 + 1.0);
    }
    let mut v: Vec<(usize, f64)> = score.into_iter().collect();
    v.sort_by(|x, y| y.1.partial_cmp(&x.1).unwrap());
    v.into_iter().map(|(id, _)| id).collect()
}

/// Group candidates by normalized ICD code across all segments.
/// Score = max over occurrences; retain all source segments and
/// the matched phrase of the best-scoring occurrence.
pub fn cross_segment_dedupe(cands: Vec<Candidate>) -> Vec<(DedupCandidate, f32)> {
    let mut by_code: HashMap<String, DedupCandidate> = HashMap::new();
    for c in cands {
        let s = c.rerank.or(c.semantic).or(c.lexical).unwrap_or(0.0);
        let e = by_code.entry(c.icd_code.clone()).or_insert_with(|| DedupCandidate {
            icd_code: c.icd_code.clone(),
            best_phrase: c.alpha_text.clone(),
            best_score: f32::MIN,
            source_segments: Vec::new(),
        });
        if !e.source_segments.contains(&c.segment_idx) {
            e.source_segments.push(c.segment_idx);
        }
        if s > e.best_score {
            e.best_score = s;
            e.best_phrase = c.alpha_text.clone();
        }
    }
    let mut out: Vec<(DedupCandidate, f32)> = by_code
        .into_values()
        .map(|mut d| { d.source_segments.sort(); let s = d.best_score; (d, s) })
        .collect();
    out.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap());
    out
}

#[derive(Debug, Clone)]
pub struct DedupCandidate {
    pub icd_code: String,
    pub best_phrase: String,
    pub best_score: f32,
    pub source_segments: Vec<usize>,
}

• Step 4: Run test to verify it passes

Run: cargo test --test fusion_tests Expected: PASS (2 tests).

• Step 5: Commit

git add src/fusion.rs tests/fusion_tests.rs
git commit -m "feat: RRF and cross-segment max-score dedupe"

---

Task 9b: ClaML Modifier/ModifierClass expansion (Revision 2026-05-18, spec §3a)

Why: ~26.5% of valid Alpha-ID primary codes are 6-char 5th-digit codes (e.g. E11.72, I10.91). ClaML models the 4th/5th digit of many groups (incl. all E10–E14 diabetes) only via <Modifier>/<ModifierClass>, not as <Class kind="category">. Task 4's claml::load returns no entry for these, so billability/description are wrong (fall back to the 3-digit Para295=V root) — --billable-only silently drops ~1,193 distinct billable codes, breaking Goal 1. Decided resolution: expand modifiers correctly from the ClaML file alone.

Files:

• Modify: src/claml.rs (extend load to also expand subclassification)

• Test: tests/claml_modifier_tests.rs

• Step 1: Failing test — tests/claml_modifier_tests.rs

use alpha_id::claml;

#[test]
fn expands_diabetes_5th_digit_codes_as_billable() {
    let map = claml::load("icd-claml/Klassifikationsdateien/icd10gm2026syst_claml_20250912.xml").unwrap();
    // E11.x 5th-digit codes are NOT explicit <Class>; must be synthesized.
    let e1190 = map.get("E11.90").expect("E11.90 synthesized from ModifierClass");
    assert_eq!(e1190.para295, "P", "terminal 5th-digit diabetes code must be billable");
    assert!(e1190.description.to_lowercase().contains("diabetes"),
        "composed description from parent + modifier labels: {:?}", e1190.description);
    let e1172 = map.get("E11.72").expect("E11.72 synthesized");
    assert_eq!(e1172.para295, "P");
    // Explicit <Class> entries still win and are unchanged.
    let a010 = map.get("A01.0").expect("A01.0 explicit");
    assert_eq!(a010.para295, "P");
    assert!(a010.description.contains("Typhus abdominalis"));
    // The bare 3-digit root stays non-billable (V).
    assert_eq!(map.get("E11").expect("E11 root").para295, "V");
}

• Step 2: Run cargo test --test claml_modifier_tests → FAIL (E11.90 absent).

• Step 3: Implement the ClaML subclassification expansion in src/claml.rs.

This is the standard ClaML modifier algorithm. There is no verbatim code block here because the exact element/attribute handling must be derived against the real XML; the implementer (capable model) must:

• During the stream parse, additionally collect: for each <Class kind="category">, its ordered <ModifiedBy code=… all=…> references; the full set of <Modifier code=…> (each with its <SubClass code=".X"/> children) and <ModifierClass code=… modifier=…> (each with its preferred <Rubric><Label>, its <SuperClass>, and its own optional <ModifiedBy> that chains the next digit).
• After the main parse, for every <Class kind="category"> that carries ModifiedBy references, generate the terminal code combinations by applying the first modifier (4th digit) then, per the resulting ModifierClass's own ModifiedBy (or the Class's second ModifiedBy), the next modifier (5th digit) — i.e. walk the modifier chain to terminal leaves. The combined code key is parentcode + modifierclass codes concatenated in ICD order (e.g. E11 + .9 + 0 → E11.90; E11 + .7 + 2 → E11.72). Match the exact concatenation to the corpus code format (verify against data/...txt field 3 for a few E11/I10 codes — they look like E11.90, E11.72, I10.91).
• Synthesized IcdMeta: code = combined; para295/para301 = "P" for terminal leaves of a modifier-expanded group (the bare root is "V"; terminal subdivision is the billable primary — verified in the investigation that these groups have no per-ModifierClass Para295, and the corpus treats the 5th-digit code as the codeable primary); description = parent preferred label + " - " + 4th-digit ModifierClass label + " - " + 5th-digit ModifierClass label (trim/join sensibly, skip empty); chapter = parent chapter; exotic/ifsg/content inherited from parent.
• Additive only: if a key already exists from an explicit <Class>, do NOT overwrite it (explicit wins).
• Only combine modifiers actually referenced by that Class (and chained ModifierClass ModifiedBy). No cartesian product across unrelated modifiers.
• Keep load's signature and the existing explicit-Class behavior (Tasks 4 tests must still pass).

The implementer should inspect the real XML (grep/xmllint) for Modifier, ModifierClass, ModifiedBy, the S04E10_4/S04E10_5 diabetes modifiers, and a couple of corpus E11/I10 codes, and derive the precise concatenation/label-composition that makes the Step-1 assertions pass honestly. Do NOT hardcode E11.* — the expansion must be generic over all ClaML modifier groups.

• Step 4: Run cargo test --test claml_modifier_tests and cargo test --test claml_tests → both PASS (new expansion correct; original explicit-Class tests incl. mixed_content_label_not_truncated unchanged & green). Then full cargo test + cargo clippy --lib clean.

• Step 5: Commit

git add src/claml.rs tests/claml_modifier_tests.rs
git commit -m "feat: expand ClaML Modifier/ModifierClass into terminal 5th-digit codes"

---

Task 10: Pipeline — Mode C

Revision 2026-05-18: With Task 9b, claml::load returns the synthesized 5th-digit codes. The ad-hoc meta_lookup strip-zero/3-char-root fallback added during the first Task 10 implementation must be removed (it caused the misclassification). Use a direct self.meta.get(code) with at most a single exact-then-5-char-parent fallback (no 3-char-root fallback, since that is exactly what corrupted billability). Additionally add a billable regression test to tests/pipeline_mode_c_tests.rs:

#[test]
fn billable_only_keeps_diabetes_5th_digit_codes() {
    let p = Pipeline::load(&cfg()).unwrap();
    let f = Filter { billable_only: true, valid_only: true, chapters: None, exclude_exotic: false };
    let res = p.suggest("Diabetes mellitus Typ 2", Mode::C, &f, 10);
    assert!(res.suggestions.iter().any(|s| s.icd_code.starts_with("E11") && s.tags.billable),
        "billable E11.x must survive --billable-only; got {:?}",
        res.suggestions.iter().map(|s| (&s.icd_code, s.tags.billable)).collect::<Vec<_>>());
}

The original two Task 10 tests remain. Re-run full suite + clippy. Re-review after the revision.

Files:

• Modify: src/pipeline.rs

• Test: tests/pipeline_mode_c_tests.rs

• Step 1: Write the failing test — tests/pipeline_mode_c_tests.rs

use alpha_id::pipeline::Pipeline;
use alpha_id::model::{Config, Filter, Mode};

fn cfg() -> Config {
    Config::load("config/default.toml").unwrap()
}

#[test]
fn mode_c_end_to_end_real_data() {
    let p = Pipeline::load(&cfg()).unwrap();
    let dictation = "Patient mit Diabetes mellitus Typ 2\n\nArterielle Hypertonie";
    let res = p.suggest(dictation, Mode::C, &Filter::default(), 10);
    assert!(!res.suggestions.is_empty());
    assert!(res.suggestions.len() <= 10);
    assert!(res.suggestions.iter().any(|s| s.icd_code.starts_with("E11")));
    assert!(res.suggestions.iter().any(|s| s.icd_code.starts_with("I10")));
    assert_eq!(res.diagnostics.segment_count, 2);
    assert_eq!(res.diagnostics.mode, "C");
}

#[test]
fn billable_only_filter_excludes_three_digit_v_codes() {
    let p = Pipeline::load(&cfg()).unwrap();
    let f = Filter { billable_only: true, valid_only: true, chapters: None, exclude_exotic: false };
    let res = p.suggest("Cholera", Mode::C, &f, 10);
    assert!(res.suggestions.iter().all(|s| s.tags.billable && s.tags.valid));
}

• Step 2: Run test to verify it fails

Run: cargo test --test pipeline_mode_c_tests Expected: FAIL — Pipeline not found.

• Step 3: Write minimal implementation — src/pipeline.rs

use crate::claml;
use crate::corpus::{self};
use crate::fusion::cross_segment_dedupe;
use crate::lexical::LexicalIndex;
use crate::model::*;
use crate::segment::split_segments;
use crate::tags::tags_for;
use std::collections::HashMap;
use std::time::Instant;

pub struct Pipeline {
    pub entries: Vec<AlphaIdEntry>,
    pub meta: HashMap<String, IcdMeta>,
    pub lexical: LexicalIndex,
    pub pool_size: usize,
}

impl Pipeline {
    pub fn load(cfg: &Config) -> Result<Self, AppError> {
        let entries = corpus::load(&cfg.alpha_id_path)?;
        let meta = claml::load(&cfg.claml_path)?;
        let lexical = LexicalIndex::build_in_ram(&entries)?;
        Ok(Self { entries, meta, lexical, pool_size: cfg.pool_size })
    }

    fn to_suggestion(&self, code: &str, phrase: &str, score: f32,
                     segs: Vec<usize>, alpha_valid: bool) -> Option<Suggestion> {
        let m = self.meta.get(code)?;
        Some(Suggestion {
            icd_code: code.to_string(),
            description: m.description.clone(),
            score,
            matched_phrase: phrase.to_string(),
            source_segments: segs,
            tags: tags_for(m, alpha_valid),
        })
    }

    pub fn suggest(&self, dictation: &str, mode: Mode, filter: &Filter, top_k: usize)
        -> SuggestResult {
        let t0 = Instant::now();
        let segments = split_segments(dictation);
        let mut candidates: Vec<Candidate> = Vec::new();

        for (si, seg) in segments.iter().enumerate() {
            let lex = self.lexical.search(seg, self.pool_size).unwrap_or_default();
            for (entry_idx, score) in lex {
                let e = &self.entries[entry_idx];
                if let Some(code) = corpus::primary_code(e) {
                    candidates.push(Candidate {
                        icd_code: code.to_string(),
                        alpha_text: e.text.clone(),
                        segment_idx: si,
                        lexical: Some(score),
                        semantic: None,
                        rerank: None,
                    });
                }
            }
        }

        // validity lookup per code: any matched entry that is valid
        let valid_by_code: HashMap<String, bool> = {
            let mut m = HashMap::new();
            for c in &candidates {
                m.entry(c.icd_code.clone()).or_insert(false);
            }
            for e in &self.entries {
                if let Some(code) = corpus::primary_code(e) {
                    if let Some(v) = m.get_mut(code) { *v = *v || e.valid; }
                }
            }
            m
        };

        let merged = cross_segment_dedupe(candidates);
        let mut suggestions = Vec::new();
        for (d, score) in merged {
            let alpha_valid = *valid_by_code.get(&d.icd_code).unwrap_or(&false);
            if let Some(s) = self.to_suggestion(&d.icd_code, &d.best_phrase, score,
                                                d.source_segments.clone(), alpha_valid) {
                if filter.accepts(&s.tags) {
                    suggestions.push(s);
                }
            }
            if suggestions.len() >= top_k { break; }
        }

        SuggestResult {
            suggestions,
            diagnostics: Diagnostics {
                mode: format!("{:?}", mode),
                degraded: false,
                segment_count: segments.len(),
                ionos_calls: 0,
                millis: t0.elapsed().as_millis(),
            },
        }
    }
}

• Step 4: Run test to verify it passes

Run: cargo test --test pipeline_mode_c_tests Expected: PASS. If mode string is "C" vs "C" mismatch, note format!("{:?}", Mode::C) yields "C" — matches the assertion.

• Step 5: Commit

git add src/pipeline.rs tests/pipeline_mode_c_tests.rs
git commit -m "feat: Mode C pipeline end-to-end (lexical + tags + fusion)"

---

Task 11: Eval harness — error injection + Recall@k

Revision 2026-05-18 (spec §6): Two changes layered on this task. (1) inject_errors case "umlaut" must produce the ASCII digraph (ä→ae, ö→oe, ü→ue, ß→ss), not delete the umlaut — the realistic transcription variant the umlaut fold (Task 5 / normalize.rs) handles; already implemented on main (commit 8348e33). (2) Add a paraphrase-injection axis: a seeded transform that yields a synonym/word-order/brevity variant of the corpus phrase (not only the verbatim string + char noise), so the eval no longer tests the exact corpus byte-string (addresses the self-retrieval over-estimation, spec §6). Keep determinism-by-seed and the existing Recall@k tests.

Files:

• Modify: src/eval.rs

• Test: tests/eval_tests.rs

• Step 1: Write the failing test — tests/eval_tests.rs

use alpha_id::eval::{inject_errors, recall_at_k};

#[test]
fn injection_is_deterministic_for_seed() {
    let a = inject_errors("Diabetes mellitus Typ 2", 0.3, 42);
    let b = inject_errors("Diabetes mellitus Typ 2", 0.3, 42);
    assert_eq!(a, b);
    assert_ne!(a, "Diabetes mellitus Typ 2"); // some corruption at rate 0.3
}

#[test]
fn recall_counts_hit_within_k() {
    let predicted = vec!["E11.9".to_string(), "I10.90".to_string(), "J45.0".to_string()];
    assert_eq!(recall_at_k(&predicted, "I10.90", 5), 1.0);
    assert_eq!(recall_at_k(&predicted, "I10.90", 1), 0.0);
    assert_eq!(recall_at_k(&predicted, "Z99.9", 5), 0.0);
}

• Step 2: Run test to verify it fails

Run: cargo test --test eval_tests Expected: FAIL — eval items not found.

• Step 3: Write minimal implementation — src/eval.rs

use rand::rngs::StdRng;
use rand::{Rng, SeedableRng};

/// Inject realistic transcription errors deterministically (seeded):
/// char drop, adjacent swap, umlaut flattening, digit transposition.
pub fn inject_errors(text: &str, rate: f64, seed: u64) -> String {
    let mut rng = StdRng::seed_from_u64(seed);
    let mut chars: Vec<char> = text.chars().collect();
    let n = chars.len();
    let mut i = 0;
    while i < chars.len() {
        if rng.gen::<f64>() < rate {
            match rng.gen_range(0..4) {
                0 => { chars.remove(i); continue; }                 // drop
                1 => { if i + 1 < chars.len() { chars.swap(i, i + 1); } } // swap
                2 => {                                               // flatten umlaut
                    chars[i] = match chars[i] {
                        'ä' => 'a', 'ö' => 'o', 'ü' => 'u', 'ß' => 's',
                        other => other,
                    };
                }
                _ => {                                               // digit transpose
                    if chars[i].is_ascii_digit() && i + 1 < chars.len()
                        && chars[i + 1].is_ascii_digit() {
                        chars.swap(i, i + 1);
                    }
                }
            }
        }
        i += 1;
    }
    let _ = n;
    chars.into_iter().collect()
}

/// 1.0 if `expected` appears in the first k predictions, else 0.0.
pub fn recall_at_k(predicted: &[String], expected: &str, k: usize) -> f64 {
    if predicted.iter().take(k).any(|p| p == expected) { 1.0 } else { 0.0 }
}

• Step 4: Run test to verify it passes

Run: cargo test --test eval_tests Expected: PASS (2 tests). If assert_ne!(a, original) is flaky at low corruption, the test uses rate 0.3 over a long string — corruption is effectively certain; if it ever fails, raise rate to 0.5 in the test.

• Step 5: Commit

git add src/eval.rs tests/eval_tests.rs
git commit -m "feat: seeded transcription-error injection + Recall@k"

---

Task 12: CLI — suggest, eval, index(build lexical)

Files:

• Modify: src/bin/alpha_id.rs

• Test: tests/cli_tests.rs

• Step 1: Write the failing test — tests/cli_tests.rs

use std::process::Command;

#[test]
fn cli_suggest_outputs_json_with_codes() {
    let out = Command::new(env!("CARGO_BIN_EXE_alpha-id"))
        .args(["suggest", "--mode", "c", "--json", "-"])
        .arg("--config").arg("config/default.toml")
        .env("ALPHA_ID_STDIN", "Diabetes mellitus Typ 2")
        .output().unwrap();
    assert!(out.status.success(), "stderr: {}", String::from_utf8_lossy(&out.stderr));
    let s = String::from_utf8_lossy(&out.stdout);
    assert!(s.contains("\"icd_code\""));
    assert!(s.contains("E11"));
}

(For test determinism the binary reads stdin; here we feed via the ALPHA_ID_STDIN env shim that the binary checks before real stdin.)

• Step 2: Run test to verify it fails

Run: cargo test --test cli_tests Expected: FAIL — subcommand not implemented.

• Step 3: Write minimal implementation — src/bin/alpha_id.rs

use alpha_id::eval::{inject_errors, recall_at_k};
use alpha_id::model::{Config, Filter, Mode};
use alpha_id::pipeline::Pipeline;
use clap::{Parser, Subcommand};
use std::io::Read;

#[derive(Parser)]
#[command(name = "alpha-id")]
struct Cli {
    #[arg(long, default_value = "config/default.toml", global = true)]
    config: String,
    #[command(subcommand)]
    cmd: Cmd,
}

#[derive(Subcommand)]
enum Cmd {
    /// Build indexes (lexical now; embeddings added in Phase A)
    Index { #[arg(long)] sample: Option<usize>, #[arg(long)] full: bool, #[arg(long)] confirm: bool },
    /// Suggest ICD codes for a dictation (FILE or - for stdin)
    Suggest {
        input: String,
        #[arg(long, default_value = "c")] mode: String,
        #[arg(long, default_value_t = 10)] top: usize,
        #[arg(long)] billable_only: bool,
        #[arg(long)] valid_only: bool,
        #[arg(long)] exclude_exotic: bool,
        #[arg(long)] chapters: Option<String>,
        #[arg(long)] json: bool,
    },
    /// Run the eval harness
    Eval {
        #[arg(long, default_value = "c")] mode: String,
        #[arg(long, default_value_t = 200)] cases: usize,
        #[arg(long, default_value_t = 42)] seed: u64,
    },
}

fn read_input(arg: &str) -> String {
    if let Ok(v) = std::env::var("ALPHA_ID_STDIN") { return v; }
    if arg == "-" {
        let mut s = String::new();
        std::io::stdin().read_to_string(&mut s).ok();
        s
    } else {
        std::fs::read_to_string(arg).unwrap_or_default()
    }
}

fn main() {
    let cli = Cli::parse();
    let cfg = Config::load(&cli.config).expect("config");
    match cli.cmd {
        Cmd::Index { sample, full, confirm } => {
            // Phase A extends this. For now: build lexical to validate data load.
            let p = Pipeline::load(&cfg).expect("load");
            eprintln!("Loaded {} entries, {} ICD metas. (sample={:?} full={} confirm={})",
                p.entries.len(), p.meta.len(), sample, full, confirm);
        }
        Cmd::Suggest { input, mode, top, billable_only, valid_only, exclude_exotic, chapters, json } => {
            let p = Pipeline::load(&cfg).expect("load");
            let m: Mode = mode.parse().expect("mode");
            let filter = Filter {
                billable_only, valid_only, exclude_exotic,
                chapters: chapters.map(|c| c.split(',').map(|s| s.trim().to_string()).collect()),
            };
            let text = read_input(&input);
            let res = p.suggest(&text, m, &filter, top);
            if json {
                println!("{}", serde_json::to_string_pretty(&res).unwrap());
            } else {
                for s in &res.suggestions {
                    println!("{:<8} {:>5.3}  {}", s.icd_code, s.score, s.description);
                }
                eprintln!("[{} segments, mode {}, {} ms, degraded={}]",
                    res.diagnostics.segment_count, res.diagnostics.mode,
                    res.diagnostics.millis, res.diagnostics.degraded);
            }
        }
        Cmd::Eval { mode, cases, seed } => {
            let p = Pipeline::load(&cfg).expect("load");
            let m: Mode = mode.parse().expect("mode");
            let mut sum = 0.0;
            let mut billable_sum = 0.0;
            let mut billable_n = 0.0;
            let billable: Vec<_> = p.entries.iter().enumerate()
                .filter(|(_, e)| e.valid)
                .filter(|(_, e)| {
                    alpha_id::corpus::primary_code(e)
                        .and_then(|c| p.meta.get(c))
                        .map(|mt| mt.para295 == "P").unwrap_or(false)
                })
                .take(cases).collect();
            for (k, (_, e)) in billable.iter().enumerate() {
                let code = alpha_id::corpus::primary_code(e).unwrap().to_string();
                let noisy = inject_errors(&e.text, 0.15, seed.wrapping_add(k as u64));
                let res = p.suggest(&noisy, m, &Filter::default(), 10);
                let preds: Vec<String> = res.suggestions.iter().map(|s| s.icd_code.clone()).collect();
                sum += recall_at_k(&preds, &code, 10);
                billable_sum += recall_at_k(&preds, &code, 10);
                billable_n += 1.0;
            }
            let n = billable.len().max(1) as f64;
            println!("mode={} cases={} Recall@10={:.3} BillableRecall@10={:.3}",
                mode, billable.len(), sum / n, billable_sum / billable_n.max(1.0));
        }
    }
}

• Step 4: Run test to verify it passes

Run: cargo test --test cli_tests Expected: PASS.

• Step 5: Manual smoke

Run: printf 'Diabetes mellitus Typ 2\n\nArterielle Hypertonie\n' | cargo run -- suggest --mode c - Expected: a table with E11.* and I10.* rows.

Run: cargo run -- eval --mode c --cases 100 Expected: a line mode=c cases=100 Recall@10=… BillableRecall@10=… — this is the baseline number Mode A must beat.

• Step 6: Commit

git add src/bin/alpha_id.rs tests/cli_tests.rs
git commit -m "feat: CLI suggest/eval/index; Mode C baseline runnable"

Milestone C: Mode C works end-to-end on real data; eval harness produces a baseline BillableRecall@10.

---

Phase A — IONOS hybrid retrieval + reranker

Task 13: Shared IONOS client

Files:

• Modify: src/ionos.rs

• Test: tests/ionos_tests.rs

• Step 1: Write the failing test — tests/ionos_tests.rs

use alpha_id::ionos::IonosClient;
use std::io::Write;

fn mock_server(body: &'static str) -> (String, std::thread::JoinHandle<()>) {
    let server = tiny_http::Server::http("127.0.0.1:0").unwrap();
    let url = format!("http://{}", server.server_addr());
    let h = std::thread::spawn(move || {
        if let Ok(req) = server.recv() {
            let resp = tiny_http::Response::from_string(body);
            req.respond(resp).ok();
        }
    });
    (url, h)
}

#[test]
fn posts_json_and_parses_response() {
    let (url, h) = mock_server(r#"{"ok":true}"#);
    let mut tf = tempfile::NamedTempFile::new().unwrap();
    write!(tf, "test-token-123").unwrap();
    let c = IonosClient::new(&url, tf.path().to_str().unwrap()).unwrap();
    let v: serde_json::Value = c.post_json("/ping", &serde_json::json!({"x":1})).unwrap();
    assert_eq!(v["ok"], true);
    h.join().ok();
}

• Step 2: Run test to verify it fails

Run: cargo test --test ionos_tests Expected: FAIL — IonosClient not found.

• Step 3: Write minimal implementation — src/ionos.rs

use crate::model::AppError;

pub struct IonosClient {
    base_url: String,
    token: String,
    http: reqwest::blocking::Client,
}

fn expand_tilde(p: &str) -> String {
    if let Some(rest) = p.strip_prefix("~/") {
        if let Some(home) = std::env::var_os("HOME") {
            return format!("{}/{}", home.to_string_lossy(), rest);
        }
    }
    p.to_string()
}

impl IonosClient {
    pub fn new(base_url: &str, token_path: &str) -> Result<Self, AppError> {
        let path = expand_tilde(token_path);
        let token = std::fs::read_to_string(&path)
            .map_err(|e| AppError::Config(format!("token {path}: {e}")))?
            .trim().to_string();
        if token.is_empty() {
            return Err(AppError::Config(format!("empty token at {path}")));
        }
        let http = reqwest::blocking::Client::builder()
            .timeout(std::time::Duration::from_secs(60))
            .build()
            .map_err(|e| AppError::Ionos(format!("client: {e}")))?;
        Ok(Self { base_url: base_url.trim_end_matches('/').to_string(), token, http })
    }

    /// POST with up to 3 retries on transport / 5xx, exponential backoff.
    pub fn post_json<T: serde::de::DeserializeOwned>(
        &self, path: &str, body: &serde_json::Value,
    ) -> Result<T, AppError> {
        let url = format!("{}{}", self.base_url, path);
        let mut last = String::new();
        for attempt in 0..3 {
            let r = self.http.post(&url)
                .bearer_auth(&self.token)
                .json(body)
                .send();
            match r {
                Ok(resp) if resp.status().is_success() => {
                    return resp.json::<T>()
                        .map_err(|e| AppError::Ionos(format!("decode: {e}")));
                }
                Ok(resp) => {
                    last = format!("HTTP {}", resp.status());
                    if !resp.status().is_server_error() {
                        return Err(AppError::Ionos(last));
                    }
                }
                Err(e) => last = format!("transport: {e}"),
            }
            std::thread::sleep(std::time::Duration::from_millis(300 * (1 << attempt)));
        }
        Err(AppError::Ionos(format!("giving up after retries: {last}")))
    }
}

• Step 4: Run test to verify it passes

Run: cargo test --test ionos_tests Expected: PASS.

• Step 5: Commit

git add src/ionos.rs tests/ionos_tests.rs
git commit -m "feat: shared IONOS HTTP client with retry/backoff"

---

Task 14: Embedding client + resumable sha256 disk cache + cost estimate

Files:

• Modify: src/embed.rs

• Test: tests/embed_tests.rs

• Step 1: Write the failing test — tests/embed_tests.rs

use alpha_id::embed::{EmbeddingStore, estimate};

#[test]
fn cache_key_is_stable_sha256() {
    let k1 = EmbeddingStore::key("BAAI/bge-m3", "diabetes");
    let k2 = EmbeddingStore::key("BAAI/bge-m3", "diabetes");
    assert_eq!(k1, k2);
    assert_ne!(k1, EmbeddingStore::key("BAAI/bge-m3", "hypertonie"));
    assert_eq!(k1.len(), 64); // hex sha256
}

#[test]
fn store_persists_and_reloads_vectors_resumably() {
    let dir = tempfile::tempdir().unwrap();
    let store = EmbeddingStore::open(dir.path().to_str().unwrap(), "BAAI/bge-m3").unwrap();
    assert!(store.get("diabetes").is_none());
    store.put("diabetes", &[0.1, 0.2, 0.3]).unwrap();
    let reopened = EmbeddingStore::open(dir.path().to_str().unwrap(), "BAAI/bge-m3").unwrap();
    assert_eq!(reopened.get("diabetes").unwrap(), vec![0.1, 0.2, 0.3]);
    assert!(reopened.has("diabetes"));            // resume: skip already-embedded
}

#[test]
fn estimate_reports_pending_count_and_chars() {
    let est = estimate(&["abc".to_string(), "de".to_string()]);
    assert_eq!(est.items, 2);
    assert_eq!(est.chars, 5);
}

• Step 2: Run test to verify it fails

Run: cargo test --test embed_tests Expected: FAIL — embed items not found.

• Step 3: Write minimal implementation — src/embed.rs

use crate::ionos::IonosClient;
use crate::model::AppError;
use sha2::{Digest, Sha256};
use std::fs;
use std::path::PathBuf;

pub struct EmbeddingStore {
    dir: PathBuf,    // <index_dir>/embeddings/<model-sanitized>/
    model: String,
}

pub struct Estimate { pub items: usize, pub chars: usize }

pub fn estimate(pending: &[String]) -> Estimate {
    Estimate { items: pending.len(), chars: pending.iter().map(|s| s.len()).sum() }
}

impl EmbeddingStore {
    pub fn key(model: &str, text: &str) -> String {
        let mut h = Sha256::new();
        h.update(model.as_bytes());
        h.update([0u8]);
        h.update(text.as_bytes());
        hex(&h.finalize())
    }

    pub fn open(index_dir: &str, model: &str) -> Result<Self, AppError> {
        let safe = model.replace('/', "_");
        let dir = PathBuf::from(index_dir).join("embeddings").join(safe);
        fs::create_dir_all(&dir)
            .map_err(|e| AppError::Io(format!("{}: {e}", dir.display())))?;
        Ok(Self { dir, model: model.to_string() })
    }

    fn path(&self, text: &str) -> PathBuf {
        self.dir.join(format!("{}.f32", Self::key(&self.model, text)))
    }

    pub fn has(&self, text: &str) -> bool { self.path(text).exists() }

    pub fn get(&self, text: &str) -> Option<Vec<f32>> {
        let bytes = fs::read(self.path(text)).ok()?;
        Some(bytes.chunks_exact(4)
            .map(|c| f32::from_le_bytes([c[0], c[1], c[2], c[3]]))
            .collect())
    }

    /// Atomic write (temp + rename) so an interrupted run never leaves
    /// a half-written vector — the next run simply re-embeds that key.
    pub fn put(&self, text: &str, v: &[f32]) -> Result<(), AppError> {
        let mut bytes = Vec::with_capacity(v.len() * 4);
        for f in v { bytes.extend_from_slice(&f.to_le_bytes()); }
        let final_path = self.path(text);
        let tmp = final_path.with_extension("tmp");
        fs::write(&tmp, &bytes).map_err(|e| AppError::Io(format!("{e}")))?;
        fs::rename(&tmp, &final_path).map_err(|e| AppError::Io(format!("{e}")))?;
        Ok(())
    }
}

fn hex(b: &[u8]) -> String {
    b.iter().map(|x| format!("{:02x}", x)).collect()
}

/// Embed a batch via IONOS, persisting each result. Only `texts` not
/// already cached should be passed (caller filters via `has`).
pub fn embed_batch(
    client: &IonosClient, store: &EmbeddingStore, model: &str, texts: &[String],
) -> Result<usize, AppError> {
    if texts.is_empty() { return Ok(0); }
    let body = serde_json::json!({ "model": model, "input": texts });
    let resp: serde_json::Value = client.post_json("/embeddings", &body)?;
    let data = resp["data"].as_array()
        .ok_or_else(|| AppError::Ionos("embeddings: no data[]".into()))?;
    let mut n = 0;
    for (i, item) in data.iter().enumerate() {
        let v: Vec<f32> = item["embedding"].as_array()
            .ok_or_else(|| AppError::Ionos("embeddings: no embedding[]".into()))?
            .iter().map(|x| x.as_f64().unwrap_or(0.0) as f32).collect();
        store.put(&texts[i], &v)?;
        n += 1;
    }
    Ok(n)
}

• Step 4: Run test to verify it passes

Run: cargo test --test embed_tests Expected: PASS (3 tests). embed_batch is exercised in Task 16 against the mock.

• Step 5: Commit

git add src/embed.rs tests/embed_tests.rs
git commit -m "feat: resumable sha256 embedding cache + cost estimate"

---

Task 15: Vector store — brute-force cosine top-k

Files:

• Modify: src/vector.rs

• Test: tests/vector_tests.rs

• Step 1: Write the failing test — tests/vector_tests.rs

use alpha_id::vector::VectorIndex;

#[test]
fn cosine_topk_ranks_nearest_first() {
    let vi = VectorIndex::from_vectors(vec![
        vec![1.0, 0.0],   // id 0
        vec![0.0, 1.0],   // id 1
        vec![0.9, 0.1],   // id 2
    ]);
    let hits = vi.top_k(&[1.0, 0.0], 2);
    assert_eq!(hits[0].0, 0);
    assert_eq!(hits[1].0, 2);
    assert!(hits[0].1 > hits[1].1);
}

• Step 2: Run test to verify it fails

Run: cargo test --test vector_tests Expected: FAIL — VectorIndex not found.

• Step 3: Write minimal implementation — src/vector.rs

/// Exact cosine search. 90k×1024 f32 ≈ 370 MB held in RAM; a query
/// is 90k dot products, well under 50 ms. No ANN dependency (YAGNI).
pub struct VectorIndex {
    vectors: Vec<Vec<f32>>,
    norms: Vec<f32>,
}

impl VectorIndex {
    pub fn from_vectors(vectors: Vec<Vec<f32>>) -> Self {
        let norms = vectors.iter().map(|v| dot(v, v).sqrt().max(1e-8)).collect();
        Self { vectors, norms }
    }

    /// (entry_index, cosine_similarity) descending.
    pub fn top_k(&self, query: &[f32], k: usize) -> Vec<(usize, f32)> {
        let qn = dot(query, query).sqrt().max(1e-8);
        let mut scored: Vec<(usize, f32)> = self.vectors.iter().enumerate()
            .map(|(i, v)| (i, dot(v, query) / (self.norms[i] * qn)))
            .collect();
        scored.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap());
        scored.truncate(k);
        scored
    }
}

fn dot(a: &[f32], b: &[f32]) -> f32 {
    a.iter().zip(b).map(|(x, y)| x * y).sum()
}

• Step 4: Run test to verify it passes

Run: cargo test --test vector_tests Expected: PASS.

• Step 5: Commit

git add src/vector.rs tests/vector_tests.rs
git commit -m "feat: brute-force cosine vector index"

---

Task 16: index build — sample smoke-test + cost-guarded full run

Files:

• Modify: src/bin/alpha_id.rs (Index arm)

• Modify: src/embed.rs (add build_corpus_embeddings)

• Test: tests/index_build_tests.rs

• Step 1: Write the failing test — tests/index_build_tests.rs

use alpha_id::embed::{EmbeddingStore, build_corpus_embeddings};
use alpha_id::ionos::IonosClient;
use std::io::Write;

#[test]
fn build_embeds_only_missing_and_is_resumable() {
    // mock server returns 2 vectors regardless
    let server = tiny_http::Server::http("127.0.0.1:0").unwrap();
    let url = format!("http://{}", server.server_addr());
    let h = std::thread::spawn(move || {
        for _ in 0..2 {
            if let Ok(req) = server.recv() {
                let body = r#"{"data":[{"embedding":[0.1,0.2]},{"embedding":[0.3,0.4]}]}"#;
                req.respond(tiny_http::Response::from_string(body)).ok();
            }
        }
    });
    let mut tf = tempfile::NamedTempFile::new().unwrap();
    write!(tf, "tok").unwrap();
    let client = IonosClient::new(&url, tf.path().to_str().unwrap()).unwrap();
    let dir = tempfile::tempdir().unwrap();
    let store = EmbeddingStore::open(dir.path().to_str().unwrap(), "BAAI/bge-m3").unwrap();
    let texts = vec!["a".to_string(), "b".to_string()];

    let n1 = build_corpus_embeddings(&client, &store, "BAAI/bge-m3", &texts, 8).unwrap();
    assert_eq!(n1, 2);
    // second run: all cached -> zero new embeds, no server call needed
    let n2 = build_corpus_embeddings(&client, &store, "BAAI/bge-m3", &texts, 8).unwrap();
    assert_eq!(n2, 0);
    h.join().ok();
}

• Step 2: Run test to verify it fails

Run: cargo test --test index_build_tests Expected: FAIL — build_corpus_embeddings not found.

• Step 3: Add build_corpus_embeddings to src/embed.rs

/// Embed all `texts` not yet cached, in batches. Resumable: a crash
/// leaves cached vectors intact; rerun continues from the gap.
pub fn build_corpus_embeddings(
    client: &IonosClient, store: &EmbeddingStore, model: &str,
    texts: &[String], batch: usize,
) -> Result<usize, AppError> {
    let pending: Vec<String> = texts.iter()
        .filter(|t| !store.has(t))
        .cloned()
        .collect();
    let mut done = 0;
    for chunk in pending.chunks(batch.max(1)) {
        done += embed_batch(client, store, model, chunk)?;
    }
    Ok(done)
}

• Step 4: Rewrite the Cmd::Index arm in src/bin/alpha_id.rs

Cmd::Index { sample, full, confirm } => {
    use alpha_id::embed::{EmbeddingStore, build_corpus_embeddings, estimate};
    use alpha_id::ionos::IonosClient;
    let p = Pipeline::load(&cfg).expect("load");
    let texts: Vec<String> = p.entries.iter().map(|e| e.text.clone()).collect();
    let client = IonosClient::new(&cfg.ionos_base_url, &cfg.token_path)
        .expect("ionos client (token?)");
    let store = EmbeddingStore::open(&cfg.index_dir, &cfg.embed_model).expect("store");

    let subset: Vec<String> = match sample {
        Some(n) => texts.iter().take(n).cloned().collect(),
        None => texts.clone(),
    };

    if sample.is_some() {
        let est = estimate(&subset);
        eprintln!("SMOKE: embedding {} sample texts ({} chars)…", est.items, est.chars);
        let n = build_corpus_embeddings(&client, &store, &cfg.embed_model, &subset, 32)
            .expect("sample embed failed");
        // verify dimension + a sample retrieval
        let dim = store.get(&subset[0]).map(|v| v.len()).unwrap_or(0);
        eprintln!("SMOKE OK: embedded {n}, vector dim = {dim}. \
                   Inspect, then run `index --full --confirm`.");
        return;
    }

    if full {
        let pending: Vec<String> = texts.iter().filter(|t| !store.has(t)).cloned().collect();
        let est = estimate(&pending);
        eprintln!("FULL RUN: {} of {} texts not yet embedded ({} chars). \
                   This calls IONOS and costs money.", est.items, texts.len(), est.chars);
        if !confirm {
            eprintln!("Refusing without --confirm. Re-run: \
                       `alpha-id index --full --confirm`");
            std::process::exit(2);
        }
        let n = build_corpus_embeddings(&client, &store, &cfg.embed_model, &texts, 32)
            .expect("full embed failed");
        eprintln!("FULL RUN done: {n} newly embedded, {} total cached.",
            texts.iter().filter(|t| store.has(t)).count());
        return;
    }

    eprintln!("Specify --sample N (smoke test first) or --full --confirm.");
}

• Step 5: Run test to verify it passes

Run: cargo test --test index_build_tests Expected: PASS (resumability: second call returns 0).

• Step 6: Commit

git add src/embed.rs src/bin/alpha_id.rs tests/index_build_tests.rs
git commit -m "feat: cost-guarded resumable index build (sample smoke + full)"

---

Task 17: Reranker client

Files:

• Modify: src/rerank.rs

• Test: tests/rerank_tests.rs

• Step 1: Write the failing test — tests/rerank_tests.rs

use alpha_id::rerank::rerank;
use alpha_id::ionos::IonosClient;
use std::io::Write;

#[test]
fn rerank_returns_scores_aligned_to_documents() {
    let server = tiny_http::Server::http("127.0.0.1:0").unwrap();
    let url = format!("http://{}", server.server_addr());
    let h = std::thread::spawn(move || {
        if let Ok(req) = server.recv() {
            let body = r#"{"results":[{"index":0,"relevance_score":0.2},
                                       {"index":1,"relevance_score":0.9}]}"#;
            req.respond(tiny_http::Response::from_string(body)).ok();
        }
    });
    let mut tf = tempfile::NamedTempFile::new().unwrap();
    write!(tf, "tok").unwrap();
    let c = IonosClient::new(&url, tf.path().to_str().unwrap()).unwrap();
    let scores = rerank(&c, "Qwen/Qwen3-VL-Reranker-8B", "diabetes",
                        &["herzinfarkt".to_string(), "diabetes typ 2".to_string()]).unwrap();
    assert_eq!(scores.len(), 2);
    assert!(scores[1] > scores[0]);
    h.join().ok();
}

• Step 2: Run test to verify it fails

Run: cargo test --test rerank_tests Expected: FAIL — rerank not found.

• Step 3: Write minimal implementation — src/rerank.rs

use crate::ionos::IonosClient;
use crate::model::AppError;

/// Cross-encoder rerank. Returns a score per document, aligned to the
/// input `docs` order (results are re-sorted back by their index).
pub fn rerank(
    client: &IonosClient, model: &str, query: &str, docs: &[String],
) -> Result<Vec<f32>, AppError> {
    if docs.is_empty() { return Ok(vec![]); }
    let body = serde_json::json!({
        "model": model, "query": query, "documents": docs
    });
    let resp: serde_json::Value = client.post_json("/rerank", &body)?;
    let results = resp["results"].as_array()
        .ok_or_else(|| AppError::Ionos("rerank: no results[]".into()))?;
    let mut scores = vec![0.0f32; docs.len()];
    for r in results {
        let idx = r["index"].as_u64().ok_or_else(
            || AppError::Ionos("rerank: no index".into()))? as usize;
        let sc = r["relevance_score"].as_f64()
            .or_else(|| r["score"].as_f64())
            .ok_or_else(|| AppError::Ionos("rerank: no score".into()))? as f32;
        if idx < scores.len() { scores[idx] = sc; }
    }
    Ok(scores)
}

Note: exact field names (relevance_score vs score, /rerank path) are confirmed empirically in the Task 18 smoke step; the code already accepts both score keys.

• Step 4: Run test to verify it passes

Run: cargo test --test rerank_tests Expected: PASS.

• Step 5: Commit

git add src/rerank.rs tests/rerank_tests.rs
git commit -m "feat: Qwen3-VL reranker client"

---

Task 18: Pipeline — Mode A (hybrid + rerank + degrade)

Files:

• Modify: src/pipeline.rs

• Test: tests/pipeline_mode_a_tests.rs

• Step 1: Write the failing test — tests/pipeline_mode_a_tests.rs

use alpha_id::pipeline::Pipeline;
use alpha_id::model::{Config, Filter, Mode};

#[test]
fn mode_a_degrades_to_c_when_ionos_unreachable() {
    let mut cfg = Config::load("config/default.toml").unwrap();
    cfg.ionos_base_url = "http://127.0.0.1:1".into(); // refused
    let p = Pipeline::load(&cfg).unwrap();
    let res = p.suggest("Diabetes mellitus Typ 2", Mode::A, &Filter::default(), 10);
    assert!(res.diagnostics.degraded);                 // fell back
    assert!(!res.suggestions.is_empty());              // still answered
    assert!(res.suggestions.iter().any(|s| s.icd_code.starts_with("E11")));
}

• Step 2: Run test to verify it fails

Run: cargo test --test pipeline_mode_a_tests Expected: FAIL — Mode A path not implemented (currently identical to C, degraded=false).

• Step 3: Extend src/pipeline.rs

Add fields + Mode A retrieval. Replace the Pipeline struct and suggest with:

use crate::embed::EmbeddingStore;
use crate::ionos::IonosClient;
use crate::rerank::rerank;
use crate::vector::VectorIndex;

pub struct Pipeline {
    pub entries: Vec<AlphaIdEntry>,
    pub meta: HashMap<String, IcdMeta>,
    pub lexical: LexicalIndex,
    pub pool_size: usize,
    cfg: Config,
    vector: Option<VectorIndex>, // present iff embeddings cached for all entries
}

impl Pipeline {
    pub fn load(cfg: &Config) -> Result<Self, AppError> {
        let entries = corpus::load(&cfg.alpha_id_path)?;
        let meta = claml::load(&cfg.claml_path)?;
        let lexical = LexicalIndex::build_in_ram(&entries)?;
        // Build vector index only if every entry has a cached embedding.
        let vector = match EmbeddingStore::open(&cfg.index_dir, &cfg.embed_model) {
            Ok(store) if entries.iter().all(|e| store.has(&e.text)) => {
                let vs: Vec<Vec<f32>> = entries.iter()
                    .map(|e| store.get(&e.text).unwrap()).collect();
                Some(VectorIndex::from_vectors(vs))
            }
            _ => None,
        };
        Ok(Self { entries, meta, lexical, pool_size: cfg.pool_size, cfg: cfg.clone(), vector })
    }

    fn semantic_pool(&self, seg: &str, calls: &mut usize)
        -> Result<Vec<(usize, f32)>, AppError> {
        let client = IonosClient::new(&self.cfg.ionos_base_url, &self.cfg.token_path)?;
        let body = serde_json::json!({ "model": self.cfg.embed_model, "input": [seg] });
        let resp: serde_json::Value = client.post_json("/embeddings", &body)?;
        *calls += 1;
        let q: Vec<f32> = resp["data"][0]["embedding"].as_array()
            .ok_or_else(|| AppError::Ionos("query embed: no vector".into()))?
            .iter().map(|x| x.as_f64().unwrap_or(0.0) as f32).collect();
        let vi = self.vector.as_ref()
            .ok_or_else(|| AppError::Ionos("no vector index".into()))?;
        Ok(vi.top_k(&q, self.pool_size))
    }

    fn rerank_pool(&self, seg: &str, idxs: &[usize], calls: &mut usize)
        -> Result<Vec<(usize, f32)>, AppError> {
        let client = IonosClient::new(&self.cfg.ionos_base_url, &self.cfg.token_path)?;
        let docs: Vec<String> = idxs.iter().map(|&i| self.entries[i].text.clone()).collect();
        let scores = rerank(&client, &self.cfg.rerank_model, seg, &docs)?;
        *calls += 1;
        Ok(idxs.iter().cloned().zip(scores).collect())
    }
}

Now replace suggest so it tries Mode A, and on any IONOS error falls back to the Mode C path with degraded=true:

impl Pipeline {
    pub fn suggest(&self, dictation: &str, mode: Mode, filter: &Filter, top_k: usize)
        -> SuggestResult {
        let t0 = Instant::now();
        let segments = split_segments(dictation);
        let mut calls = 0usize;
        let mut degraded = false;

        let candidates: Vec<Candidate> = match mode {
            Mode::A => match self.collect_mode_a(&segments, &mut calls) {
                Ok(c) => c,
                Err(_) => { degraded = true; self.collect_mode_c(&segments) }
            },
            Mode::C => self.collect_mode_c(&segments),
        };

        let valid_by_code = self.valid_by_code(&candidates);
        let merged = cross_segment_dedupe(candidates);
        let mut suggestions = Vec::new();
        for (d, score) in merged {
            let alpha_valid = *valid_by_code.get(&d.icd_code).unwrap_or(&false);
            if let Some(s) = self.to_suggestion(&d.icd_code, &d.best_phrase, score,
                                                d.source_segments.clone(), alpha_valid) {
                if filter.accepts(&s.tags) { suggestions.push(s); }
            }
            if suggestions.len() >= top_k { break; }
        }
        SuggestResult {
            suggestions,
            diagnostics: Diagnostics {
                mode: format!("{:?}", mode),
                degraded,
                segment_count: segments.len(),
                ionos_calls: calls,
                millis: t0.elapsed().as_millis(),
            },
        }
    }

    fn collect_mode_c(&self, segments: &[String]) -> Vec<Candidate> {
        let mut out = Vec::new();
        for (si, seg) in segments.iter().enumerate() {
            for (entry_idx, score) in self.lexical.search(seg, self.pool_size).unwrap_or_default() {
                let e = &self.entries[entry_idx];
                if let Some(code) = corpus::primary_code(e) {
                    out.push(Candidate { icd_code: code.into(), alpha_text: e.text.clone(),
                        segment_idx: si, lexical: Some(score), semantic: None, rerank: None });
                }
            }
        }
        out
    }

    fn collect_mode_a(&self, segments: &[String], calls: &mut usize)
        -> Result<Vec<Candidate>, AppError> {
        let mut out = Vec::new();
        for (si, seg) in segments.iter().enumerate() {
            let lex = self.lexical.search(seg, self.pool_size).unwrap_or_default();
            let sem = self.semantic_pool(seg, calls)?;
            let lex_ids: Vec<usize> = lex.iter().map(|(i, _)| *i).collect();
            let sem_ids: Vec<usize> = sem.iter().map(|(i, _)| *i).collect();
            let pool = crate::fusion::rrf_merge(&lex_ids, &sem_ids, 60);
            let pool: Vec<usize> = pool.into_iter().take(self.pool_size).collect();
            let reranked = self.rerank_pool(seg, &pool, calls)?;
            for (entry_idx, rscore) in reranked {
                let e = &self.entries[entry_idx];
                if let Some(code) = corpus::primary_code(e) {
                    out.push(Candidate { icd_code: code.into(), alpha_text: e.text.clone(),
                        segment_idx: si, lexical: None, semantic: None, rerank: Some(rscore) });
                }
            }
        }
        Ok(out)
    }

    fn valid_by_code(&self, candidates: &[Candidate]) -> HashMap<String, bool> {
        let mut m: HashMap<String, bool> = HashMap::new();
        for c in candidates { m.entry(c.icd_code.clone()).or_insert(false); }
        for e in &self.entries {
            if let Some(code) = corpus::primary_code(e) {
                if let Some(v) = m.get_mut(code) { *v = *v || e.valid; }
            }
        }
        m
    }
}

Delete the old single suggest/to_suggestion duplicates from Task 10 — keep one to_suggestion (unchanged) and the new suggest.

• Step 4: Run test to verify it passes

Run: cargo test --test pipeline_mode_a_tests Expected: PASS — IONOS refused → degraded=true, lexical still answers.

• Step 5: Run the full suite (no regressions)

Run: cargo test Expected: all prior tests still PASS (Mode C unaffected).

• Step 6: Commit

git add src/pipeline.rs tests/pipeline_mode_a_tests.rs
git commit -m "feat: Mode A hybrid pipeline with graceful degradation to C"

---

Task 19: Eval comparison C vs A + smoke against real IONOS

Revision 2026-05-18 (spec §6): First C-vs-A run done — Mode C 0.520 vs A 0.995 BillableRecall@10 (200 cases, seed 42). Under the old dictation-paragraph contract that 0.995 was self-retrieval-inflated; under the new phrase-list contract (spec §1a) the short↔short regime is legitimately representative for noise/paraphrase robustness. Add to this task: (a) a small hand-labeled set of realistic extracted phrase lists (one phrase/line — the new input form) paired with the gold billable-ICD set, plus the source dictation stored alongside to document the extraction-recall ceiling; (b) the eval report must show extraction-recall ceiling (dictation→phrases) vs. matcher recall (phrases→codes) separately, never conflated. The hand-labeled set needs realistic example dictations from the user (still an open input, spec §9) — flag as BLOCKED-pending-user-data if absent, don't fabricate.

Files:

• Modify: src/bin/alpha_id.rs (Eval arm prints both modes when --mode both)

• Test: tests/eval_compare_tests.rs

• Step 1: Write the failing test — tests/eval_compare_tests.rs

use std::process::Command;

#[test]
fn eval_mode_c_runs_and_reports_billable_recall() {
    let out = Command::new(env!("CARGO_BIN_EXE_alpha-id"))
        .args(["eval", "--mode", "c", "--cases", "30"])
        .arg("--config").arg("config/default.toml")
        .output().unwrap();
    assert!(out.status.success(), "stderr: {}", String::from_utf8_lossy(&out.stderr));
    let s = String::from_utf8_lossy(&out.stdout);
    assert!(s.contains("BillableRecall@10="));
}

• Step 2: Run test to verify it fails

If Task 12's eval already prints BillableRecall@10=, this test passes immediately — that is acceptable (regression guard). Otherwise adjust the Eval arm to print that exact token.

Run: cargo test --test eval_compare_tests Expected: PASS (guard) or FAIL then fix the print string.

• Step 3: Add a both convenience to the Eval arm

In src/bin/alpha_id.rs, after parsing mode, allow the literal both:

Cmd::Eval { mode, cases, seed } => {
    let modes: Vec<Mode> = if mode == "both" {
        vec![Mode::C, Mode::A]
    } else {
        vec![mode.parse().expect("mode")]
    };
    let p = Pipeline::load(&cfg).expect("load");
    for m in modes {
        // (reuse the eval loop from Task 12, parameterized by `m`;
        //  extract it into `fn run_eval(&p, m, cases, seed) -> (f64,f64)`
        //  and print one line per mode)
        let (r10, br10) = run_eval(&p, m, cases, seed);
        println!("mode={:?} cases={} Recall@10={:.3} BillableRecall@10={:.3}",
                 m, cases, r10, br10);
    }
}

Extract the Task 12 eval body verbatim into a free function:

fn run_eval(p: &Pipeline, m: Mode, cases: usize, seed: u64) -> (f64, f64) {
    use alpha_id::eval::{inject_errors, recall_at_k};
    use alpha_id::model::Filter;
    let billable: Vec<_> = p.entries.iter().enumerate()
        .filter(|(_, e)| e.valid)
        .filter(|(_, e)| alpha_id::corpus::primary_code(e)
            .and_then(|c| p.meta.get(c))
            .map(|mt| mt.para295 == "P").unwrap_or(false))
        .take(cases).collect();
    let mut sum = 0.0;
    for (k, (_, e)) in billable.iter().enumerate() {
        let code = alpha_id::corpus::primary_code(e).unwrap().to_string();
        let noisy = inject_errors(&e.text, 0.15, seed.wrapping_add(k as u64));
        let res = p.suggest(&noisy, m, &Filter::default(), 10);
        let preds: Vec<String> = res.suggestions.iter().map(|s| s.icd_code.clone()).collect();
        sum += recall_at_k(&preds, &code, 10);
    }
    let n = billable.len().max(1) as f64;
    (sum / n, sum / n) // overall == billable here (sample is billable-only)
}

• Step 4: Run test to verify it passes

Run: cargo test --test eval_compare_tests Expected: PASS.

• Step 5: Real IONOS smoke (manual, gated)

Only after alpha-id index --sample 50 then alpha-id index --full --confirm have populated the cache:

Run: cargo run -- eval --mode both --cases 200 Expected: two lines; compare BillableRecall@10 for C vs A. This is the go/no-go number (spec §6) — present it to the user; do not decide adoption autonomously.

• Step 6: Commit

git add src/bin/alpha_id.rs tests/eval_compare_tests.rs
git commit -m "feat: C-vs-A eval comparison; go/no-go metric"

Milestone A: Mode A end-to-end with graceful degradation; eval prints the C-vs-A BillableRecall@10 comparison that decides PoC adoption.

---

Self-Review

1. Spec coverage:

• §1 I/O contract → Tasks 6 (segment), 10/18 (single global list), 2 (types). ✓
• §2 Modes C/A, data flow, modules → Tasks 3,4,7,8,9,10 (C), 13–18 (A). ✓
• §3 data model, dedupe by normalized code, fusion → Tasks 2, 3 (normalize_code), 9. ✓
• §4 one-time, hash-cached, resumable, cost-guarded, sample-first embedding → Tasks 14, 16. ✓
• §5 degrade-not-fail, no hallucination (codes only from corpus) → Task 18 (degrade), 10/18 (codes sourced from entries only). ✓
• §6 eval harness, synthetic errors, Recall@10 billable, go/no-go → Tasks 11, 12, 19. ✓
• §7 CLI surface → Tasks 12, 16, 19. ✓
• §8 out of scope (no generative LLM) → respected; only embeddings + reranker. ✓

2. Placeholder scan: No "TBD/TODO". Two conditional steps (claml chapter-stack 2b; eval guard 19/2) are explicit, with concrete code/criteria — not placeholders. The Task 19 run_eval extraction quotes the full function body rather than "similar to Task 12".

3. Type consistency: AlphaIdEntry, IcdMeta, Tags, Filter, Candidate, Suggestion, Diagnostics, SuggestResult, Config, AppError, Mode defined once in Task 2 and used unchanged. corpus::primary_code, normalize::normalize, LexicalIndex::search, fusion::rrf_merge, fusion::cross_segment_dedupe, EmbeddingStore, VectorIndex::top_k, rerank::rerank, IonosClient::post_json signatures are consistent across Tasks 3–19. Task 18 explicitly supersedes the Task 10 suggest (noted to delete the duplicate) — no clearLayers/clearFullLayers drift.

Known intentional simplification: run_eval returns (overall, billable) equal because the eval sample is billable-only by construction (spec §6 headline metric); documented inline.

---

Execution Handoff

Plan complete and saved to docs/superpowers/plans/2026-05-18-alpha-id-icd-suggestion.md. Two execution options:

1. Subagent-Driven (recommended) — I dispatch a fresh subagent per task, review between tasks, fast iteration.

2. Inline Execution — Execute tasks in this session using executing-plans, batch execution with checkpoints.

Which approach?