give gpt_oss_120b the safe (flag-dont-guess) system prompt #18

2026-07-17T23:40:06+02:00

claude commented

2026-07-17 23:40:06 +02:00

What

Replace server/prompts/default_system_prompt.md with the llama prompt content, so the default backend gpt_oss_120b (which carries all real traffic) uses the safe "keep the dictated word and flag it — never guess" prompt instead of the permissive "substitute the term and bracket the original" one.

Why

Analysis of 16 real alpha-test cases showed the default prompt systematically over-corrects: it substitutes dictated medical terms with the model's own guess and, when the guess is wrong, inverts meaning, deletes negations, or invents drugs/doses (e.g. kontrastmittelaffin→kontrastmittelfrei, keine relevanten Vitien→Stenosen, invented Novalgin 500 mg).

Evidence

Experiments sandbox, full pipeline (pre-gazetteer → gpt-oss-120b → post-gazetteer), 3 runs per case, each divergence independently judged:

prompt	runs with a dangerous error	runs with no output (failed)
old default	20 / 45 (44%)	3
this (llama) prompt	5 / 49 (10%)	0

Same model, same gazetteer — only the prompt changed. No case-specific vocabulary was baked into the prompt.

Residual (honest)

Not zero: one hard ASR case (davon→Tavor) still defeats both prompts and points to a separate gazetteer/model-prior follow-up.

Follow-up (not in this PR)

The DEFAULT_SYSTEM_PROMPT doc-comment in analyze/backend.rs still describes the old prompt's design and c414cf52 validation; it should be refreshed. The two prompt files are now content-identical (as the backend.rs:61 comment already anticipated).

Tests

Full server suite green (lib 233 passed / 0 failed; all integration suites green). No test depended on the prompt's exact text.

## What Replace `server/prompts/default_system_prompt.md` with the llama prompt content, so the default backend `gpt_oss_120b` (which carries all real traffic) uses the safe *"keep the dictated word and flag it — never guess"* prompt instead of the permissive *"substitute the term and bracket the original"* one. ## Why Analysis of 16 real alpha-test cases showed the default prompt systematically over-corrects: it substitutes dictated medical terms with the model's own guess and, when the guess is wrong, inverts meaning, deletes negations, or invents drugs/doses (e.g. `kontrastmittelaffin`→`kontrastmittelfrei`, `keine relevanten Vitien`→`Stenosen`, invented `Novalgin 500 mg`). ## Evidence Experiments sandbox, full pipeline (pre-gazetteer → `gpt-oss-120b` → post-gazetteer), 3 runs per case, each divergence independently judged: | prompt | runs with a dangerous error | runs with no output (failed) | |---|---|---| | old default | 20 / 45 (44%) | 3 | | this (llama) prompt | 5 / 49 (10%) | 0 | Same model, same gazetteer — only the prompt changed. No case-specific vocabulary was baked into the prompt. ## Residual (honest) Not zero: one hard ASR case (`davon`→`Tavor`) still defeats both prompts and points to a separate gazetteer/model-prior follow-up. ## Follow-up (not in this PR) The `DEFAULT_SYSTEM_PROMPT` doc-comment in `analyze/backend.rs` still describes the *old* prompt's design and c414cf52 validation; it should be refreshed. The two prompt files are now content-identical (as the `backend.rs:61` comment already anticipated). ## Tests Full server suite green (lib 233 passed / 0 failed; all integration suites green). No test depended on the prompt's exact text.

claude added 1 commit 2026-07-17 23:40:07 +02:00

fix(analyze): give gpt_oss_120b the safe (flag-don't-guess) system prompt 1a87c6b5ea

Replace default_system_prompt.md with the llama prompt content. The default
backend (gpt_oss_120b) carries all real traffic, but used the permissive
prompt that lets the model SUBSTITUTE dictated terms with its own guess;
the llama prompt keeps the dictated word and only flags it for review.

Validated in the experiments sandbox against 16 real alpha-test cases
(full pipeline: pre-gazetteer -> gpt_oss_120b -> post-gazetteer, 3 runs
each, independently judged): dangerous-error rate per run drops from
44% (20/45) to 10% (5/49), and the 3 runs that produced no output at all
(reasoning length-exhaustion) drop to 0.

The two prompt files are now content-identical, as backend.rs already
anticipated. Follow-up: the DEFAULT_SYSTEM_PROMPT doc-comment in
analyze/backend.rs still describes the old prompt and should be refreshed.

claude added 1 commit 2026-07-18 14:42:33 +02:00

docs(analyze): refresh DEFAULT_SYSTEM_PROMPT doc-comments after prompt swap 65a84655a9

The doc-comments still described the old permissive prompt (the c414cf52
sandbox validation, the "AKTIVE KORREKTUR" rationale, the old block layout).
Rewrite them to describe the current "flag, don't guess" prompt and to note
that DEFAULT_SYSTEM_PROMPT and LLAMA_SYSTEM_PROMPT are now content-identical.

claude changed title from ~~WIP: give gpt_oss_120b the safe (flag-don't-guess) system prompt~~ to give gpt_oss_120b the safe (flag-dont-guess) system prompt

2026-07-18 14:42:44 +02:00

claude merged commit 65a84655a9 into main

2026-07-18 14:43:03 +02:00

claude deleted branch worktree-swap-default-prompt

2026-07-18 14:43:03 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Brummel/doctate#18