feat: multi-backend LLM layer with per-request choice on case page

Replace the single [llm] block in settings.toml with a curated catalog
of LLM "backends" (provider + model + sampling params + system prompt)
in server/src/analyze/backend.rs. Two backends ship initially:
gpt_oss_120b (default, reasoning_effort=medium) and llama_3_1_405b
(uses response_format: json_schema to sidestep the <|eot_id|> leak).

The case page now renders one submit button per available backend; the
clicked button's name=backend value rides through AnalysisInput.backend_id
to the worker, which looks it up via find_backend() with default fallback.
A submitted unknown backend id is rejected as 400. .analysis_failed.json
records the failed backend and the failure banner surfaces its label.

The API key now comes from IONOS_API_KEY (env), not settings.toml; the
[llm] section is read into a discarded field so old configs still load.
Backends without a satisfied requires_api_key are filtered from the UI
(any_backend_available()). Three end-to-end tests that mocked the LLM
endpoint via settings.llm.url are #[ignore]'d with a clear note — they
need per-test backend injection (Arc<Vec<LlmBackend>> through the worker)
before they can be re-enabled.
This commit is contained in:
2026-05-03 14:57:21 +02:00
parent cbb072d0cc
commit 23ef84d9d8
18 changed files with 596 additions and 307 deletions
+35 -10
View File
@@ -8,11 +8,34 @@
use std::collections::HashMap;
use std::path::PathBuf;
use std::sync::Arc;
use std::sync::{Arc, OnceLock};
use doctate_server::config::{Config, User};
use doctate_server::settings::Settings;
/// Make a (test) `IONOS_API_KEY` visible before the static backend catalog
/// in `analyze::backend` reads it. Idempotent — only the first call writes
/// the env var; later calls are a no-op. Tests that exercise paths gated
/// by `analyze::backend::any_backend_available()` should call this (the
/// builder does it automatically when `with_llm` is invoked).
///
/// SAFETY: in Edition 2024, `std::env::set_var` is `unsafe` because it
/// races with concurrent `getenv` on POSIX. The `OnceLock` guarantees
/// that the write happens at most once, before any test thread has had
/// a chance to read the env via `analyze::backend::backends()`. As long
/// as `ensure_test_llm_env()` is called in `TestConfig::new()` (i.e. at
/// the top of every test setup), no other test thread observes the
/// env-var being mutated mid-flight.
pub fn ensure_test_llm_env() {
static ONCE: OnceLock<()> = OnceLock::new();
ONCE.get_or_init(|| {
// SAFETY: see function docstring — single-threaded init under OnceLock.
unsafe {
std::env::set_var("IONOS_API_KEY", "test-key");
}
});
}
/// Unique tempdir path for a given label. `process::id()` + UUID v4
/// keeps parallel `cargo test` runs from colliding; the label lets
/// humans identify which test owns which leftover directory when
@@ -50,6 +73,9 @@ impl TestConfig {
/// Start a fresh builder. `label` is used only to name the tempdir
/// when no explicit `data_path` is set via `with_data_path`.
pub fn new() -> Self {
// Pre-seed the test API key so any code path that touches the
// backend catalog observes a non-empty `IONOS_API_KEY`.
ensure_test_llm_env();
Self {
data_path: None,
users: Vec::new(),
@@ -152,15 +178,14 @@ impl TestConfig {
});
let mut settings = Settings::default();
if let Some(v) = self.llm_url {
settings.llm.url = v;
}
if let Some(v) = self.llm_api_key {
settings.llm.api_key = v;
}
if let Some(v) = self.llm_model {
settings.llm.model = v;
}
// LLM-related fields used to live on `Settings`; with the multi-backend
// refactor they are now owned by `analyze::backend`. The `with_llm*`
// builder methods are kept for source-compat but their stored URL/
// api_key/model values no longer flow into `settings`. Tests that
// rely on a working LLM mock must inject backends through other
// means (currently: those tests are #[ignore]'d, see the test
// attribute comment).
let _ = (self.llm_url, self.llm_api_key, self.llm_model);
if let Some(v) = self.whisper_url {
settings.whisper.url = v;
}