feat: multi-backend LLM layer with per-request choice on case page
Replace the single [llm] block in settings.toml with a curated catalog of LLM "backends" (provider + model + sampling params + system prompt) in server/src/analyze/backend.rs. Two backends ship initially: gpt_oss_120b (default, reasoning_effort=medium) and llama_3_1_405b (uses response_format: json_schema to sidestep the <|eot_id|> leak). The case page now renders one submit button per available backend; the clicked button's name=backend value rides through AnalysisInput.backend_id to the worker, which looks it up via find_backend() with default fallback. A submitted unknown backend id is rejected as 400. .analysis_failed.json records the failed backend and the failure banner surfaces its label. The API key now comes from IONOS_API_KEY (env), not settings.toml; the [llm] section is read into a discarded field so old configs still load. Backends without a satisfied requires_api_key are filtered from the UI (any_backend_available()). Three end-to-end tests that mocked the LLM endpoint via settings.llm.url are #[ignore]'d with a clear note — they need per-test backend injection (Arc<Vec<LlmBackend>> through the worker) before they can be re-enabled.
This commit is contained in:
@@ -23,31 +23,8 @@ url = "http://localhost:11434"
|
||||
model = "gemma3:4b"
|
||||
keep_alive = 0
|
||||
|
||||
# LLM provider: swap url/api_key/model between blocks to switch providers.
|
||||
# Only one [llm] block may be active at a time — comment the others.
|
||||
|
||||
# Example: Ionos hosted
|
||||
[llm]
|
||||
url = "https://openai.inference.de-txl.ionos.com"
|
||||
api_key = ""
|
||||
model = ""
|
||||
temperature = 0
|
||||
timeout_seconds = 180
|
||||
# Single line or TOML triple-string for multi-line prompts.
|
||||
system_prompt = """
|
||||
Du bist ein medizinischer Assistent. Fasse die folgenden diktierten Abschnitte
|
||||
zu einem zusammenhängenden Text zusammen. Bereinigen und strukturieren, nichts
|
||||
hinzufügen, nichts interpretieren, keine Diagnose, keine Arztbrief-Struktur.
|
||||
Entferne Redundanzen. Spätere Aufnahmen haben Vorrang — Korrekturen, Nachträge
|
||||
und Widersprüche zugunsten der chronologisch letzten Aussage auflösen.
|
||||
Antworte auf Deutsch. Nur der zusammengefasste Text, keine Einleitung, kein
|
||||
Abschlusssatz.
|
||||
"""
|
||||
|
||||
# Example: Ollama local
|
||||
# [llm]
|
||||
# url = "http://localhost:11434"
|
||||
# api_key = ""
|
||||
# model = "SimonPu/gpt-oss:20b_Q4_K_M"
|
||||
# temperature = 0
|
||||
# timeout_seconds = 300
|
||||
# [llm] is no longer read.
|
||||
# LLM backends are defined in code (server/src/analyze/backend.rs); the API
|
||||
# key comes from the IONOS_API_KEY env var (or the equivalent for other
|
||||
# providers). An old [llm] block may stay in this file — it is parsed and
|
||||
# discarded for backwards compatibility, but values inside have no effect.
|
||||
|
||||
Reference in New Issue
Block a user