This commit replaces the disk-based `analysis_input.json` marker for
in-progress analysis jobs with an in-memory
`Arc<RwLock<HashSet<PathBuf>>>`.
Previously, the existence of `analysis_input.json` was used as a signal
that a case was queued or being analyzed. This led to stale "Queued"
states after server restarts, as the disk file would persist while the
server process tracking it had vanished.
The new `InFlight` struct, held in `AppState`, provides a process-local
marker. This ensures that:
- Server restarts correctly reset the state, as the `HashSet` is
cleared.
- Race conditions for triggering analysis are handled by the `try_claim`
method, replacing the previous reliance on file system `O_EXCL` flags.
- The `AnalyzeJob` payload now travels with the job over the channel,
eliminating the need for workers to re-read `analysis_input.json` from
disk.
This change simplifies state management and improves the reliability of
analysis job tracking.
The `boot_time` parameter is introduced to `evaluate_state` and its
callers. This parameter acts as a lower bound for the "seen at"
timestamp used in determining the `Idle` state for cases.
Previously, the `elapsed` duration was calculated as
`now.duration_since(latest_mtime)`. This meant that a server restart
could immediately cause old recordings to be classified as `Idle` if
their modification times were significantly in the past, potentially
triggering a large number of LLM calls upon the first render of the
`/cases` page.
By using `seen_at = std::cmp::max(latest_mtime, boot_time)`, the idle
calculation now ensures that a fresh server start respects a grace
period. Old recordings will not satisfy the idle threshold until the
elapsed time since the server's boot (or the recording's modification
time, whichever is later) exceeds the `idle_threshold`.
`SystemTime::UNIX_EPOCH` is used as the default `boot_time` in tests and
in `create_router_and_session_store_with_settings`. This preserves
existing test behavior where the idle grace period is effectively
disabled. For production, `main.rs` now captures the server's actual
boot time and passes it down.
This change also refactors `evaluate_case` and `try_enqueue` to accept
the new `boot_time` parameter. `try_enqueue_all_for_user` is updated to
pass `boot_time` along. `boot_scan_state` and `compute_state_for_user`
in `recovery.rs` also receive `boot_time`.
Additionally, a new test case
`analyze_required_endpoint_enqueues_only_required_cases` is added to
verify the behavior of the `/cases/analyze-required` endpoint. This
endpoint now correctly enqueues only `Required`
The `document.md` file has been replaced by `document.json`. This new
format acts as a JSON envelope, containing the original markdown content
along with metadata such as `covered_mtime`, `written_at`, and
`backend_id`.
This change addresses a bug where new recordings arriving during an LLM
analysis call could be ignored. The `covered_mtime` field in the new
JSON envelope ensures that the document's metadata accurately reflects
the state of recordings at the time of analysis, preventing incorrect
"analysis current" states.
Additionally, the `analysis_input.json` file is now removed upon
successful analysis, and a failure marker is used to indicate analysis
failures. This ensures that a case is re-analyzed if necessary.
The `analysis_idle_threshold_secs` configuration option has been
introduced to control the delay before a case in the `Required` state is
automatically promoted to `Idle` and enqueued for LLM analysis.
The /web/ prefix predated the /api/ split; today it just clutters every URL
without disambiguating anything. All 16 browser routes move to the apex
(/cases, /login, /magic, /events, /audio/...). The 6 /api/* routes are
unchanged. A new /->>/cases redirect closes the apex 404.
The open-redirect guard in magic.rs and case_actions.rs flips from a
positive whitelist (starts_with("/web/")) to a deny-list: same-origin path,
not protocol-relative, not under /api/, no \. The /api/ exclusion is now
load-bearing and covered by tests.
Pre-production: no transition redirects.
Introduce `display_name` field to `User` struct for human-readable names
in the UI.
This falls back to the user's `slug` if not provided.
Update `AuthenticatedWebUser` and `MyCasesTemplate` to use this new
field for rendering.
Add tests for parsing `display_name` and ensure fallback logic works
correctly.
Update `users.toml.example` to document the new field.
Replace the single [llm] block in settings.toml with a curated catalog
of LLM "backends" (provider + model + sampling params + system prompt)
in server/src/analyze/backend.rs. Two backends ship initially:
gpt_oss_120b (default, reasoning_effort=medium) and llama_3_1_405b
(uses response_format: json_schema to sidestep the <|eot_id|> leak).
The case page now renders one submit button per available backend; the
clicked button's name=backend value rides through AnalysisInput.backend_id
to the worker, which looks it up via find_backend() with default fallback.
A submitted unknown backend id is rejected as 400. .analysis_failed.json
records the failed backend and the failure banner surfaces its label.
The API key now comes from IONOS_API_KEY (env), not settings.toml; the
[llm] section is read into a discarded field so old configs still load.
Backends without a satisfied requires_api_key are filtered from the UI
(any_backend_available()). Three end-to-end tests that mocked the LLM
endpoint via settings.llm.url are #[ignore]'d with a clear note — they
need per-test backend injection (Arc<Vec<LlmBackend>> through the worker)
before they can be re-enabled.
This commit removes the `whisper_variant` and `whisper_hotwords_variant`
fields from the `run_id` generation and the `print_meta_summary`
function.
These variants are no longer used as the project is shifting focus to
LLM-based generation. The `run_full_case.rs` example has also been
updated to reflect this change.
This commit introduces the `Loudness` struct to store replay-gain data
(mean, max, and calculated gain in dB) derived from `ffmpeg`'s
`volumedetect`.
The `RecordingMeta` struct is updated to include an optional `loudness`
field. This allows for consistent playback volume across recordings with
varying microphone levels by applying gain in the browser.
Additionally, new `server/src/loudness.rs` and
`server/src/transcribe/ffmpeg.rs` modules are added. The former defines
constants for replay-gain targets and logic to compute gain, while the
latter handles audio analysis using `ffmpeg` to extract loudness
metrics.
The `#[serde(default)]` attribute on the `loudness` field in
`RecordingMeta` ensures backward compatibility with older metadata files
that do not contain this field. Tests are included to verify round-trip
serialization and parsing of older formats.
This commit changes the way transcriptions are stored and accessed.
Instead of using plain text files (`.transcript.txt`), transcriptions
will now be part of a JSON metadata file (`<stem>.json`). This allows
for richer metadata to be stored alongside the transcript, such as
duration, and provides a more robust mechanism for tracking
transcription states.
The changes include:
- Updating documentation and code to reflect the new `.json` file
extension.
- Modifying file handling logic to read and write JSON metadata.
- Adjusting tests to accommodate the new file format.
Replaces the `.transcript.txt` sidecar with a structured `.json` file
for recording metadata. This change consolidates transcript text,
duration, and other potential metadata into a single, extensible JSON
object.
This also refactors the `TranscriptState` enum to better represent the
on-disk state (absence of file means pending) and the in-memory
representation. The `Transcript` enum now specifically models the
terminal outcomes of the transcriber (`Silent` or `Content`).
The commit includes updates to documentation, data structures, path
handling, and various tests to align with the new metadata format.
Config sheds its 14 Bucket-B fields (retention, whisper, ollama, llm) and
gains a sibling Arc<Settings> in AppState, loaded from settings.toml at
startup via Settings::load_or_default. Workers (transcribe, analyze,
recovery, auto_trigger) and routes (health, bulk, case_actions,
user_web) take settings as a separate parameter or State<> extractor;
Config::llm_configured moves to Settings::llm_configured.
TestConfig::build_pair() returns (Arc<Config>, Arc<Settings>); the new
create_router_with_settings / create_router_and_session_store_with_settings
helpers wire both into integration tests that exercise LLM/Whisper/Ollama.
The legacy single-Arc create_router falls back to Settings::default()
so the 17 non-LLM tests stay untouched.
Verified: cargo build clean, clippy --all-targets clean, 318 tests pass.
Adds a new POST endpoint for web-based oneliner overrides and updates
the UI to allow manual editing. The person icon replaces the pencil icon
for doctor-authored titles, signifying manual authorship rather than
editability.
Pulls three pattern groups into shared helpers so a rename or wire-format
tweak edits one file instead of 10+:
- BulkAction enum in doctate-common replaces the "close"/"analyze"/"reset"
string literals that were duplicated between the bulk handler and 3
attack/CSRF test files. FromStr preserves the exact "Unbekannte Aktion"
error shape; the handler match is now exhaustive over the enum.
- join_url helper in doctate-common absorbs 7 identical
trim_end_matches+format! call sites across client-core, client-desktop,
server/transcribe (ollama, whisper), and server/analyze (llm).
- server/tests/common/artefacts.rs re-exports ONELINER_FILENAME
(doctate-common), DOCUMENT_FILE + ANALYSIS_INPUT_FILE
(doctate-server::analyze), and CLOSE_MARKER (doctate-server::paths) so
test files reference the canonical name instead of inlining literals.
Also lifts client-desktop local duplication:
- paths.rs: project_path helper collapses 4 identical ProjectDirs chains
- main.rs: or_die helper replaces 4 eprintln!+exit(1) blocks
- app.rs: named RecordingContext struct replaces (Uuid, String) tuple
at 3 sites around the ffmpeg-flush finalization path
Verification: 397 tests pass (baseline was 389; +8 for new unit tests on
BulkAction and join_url), 0 failed, 4 ignored (unchanged). clippy clean.
Finishes the lift of shared helpers into `tests/common/`. Covered:
- health, web, oneliners_api, upload (31 tests; upload exercises the
new `multipart_upload_body` helper driven by doctate_common field
constants)
- sse_integration, sse_cleanup, transcribe, oneliner_heal_decoupled,
silent_case_empty, failed_only_case_empty_oneliner,
transient_failure_retries (24 tests)
Also applied cargo fmt across the test tree and fixed one clippy
needless_borrows_for_generic_args warning in analyze_test.
All 387 tests pass; 3 ignored (as before).
Side effect: health_test previously used a hardcoded `/tmp/doctate-test`
data path, which parallel `cargo test` runs could collide on. The
migration replaces it with the common unique-tmpdir pattern, removing
a latent flake.
Lift duplicated test-harness code (User factories, TestConfig builder,
login/CSRF flow, form/json helpers, case-directory seeding, URL path
builders) into `server/tests/common/` so future API changes touch one
file instead of every integration test.
Migrated batches: csrf_attack, auth, login, magic_link, security_headers
(28 tests); analyze, case_page, delete_recording, close_watermark,
retention_sweep (67 tests). All 95 tests still green.
Net line delta across these 10 files: +1113 / -1896 (~780 lines removed),
plus ~500 lines of new shared infrastructure under tests/common/.
Brummel