Feat: Add gazetteer normalization to transcription
This commit integrates the Gazetteer into the transcription pipeline to normalize terminology. The Gazetteer, loaded at startup, is now passed to the transcription worker. Before persisting the transcription text, it is processed through the Gazetteer's `replace` method, ensuring standardized terminology. This normalization is also applied to the generated "oneliner" text. Additionally, the `regenerate_missing_oneliners` function in the recovery module has been updated to accept and utilize the Gazetteer, ensuring that regenerated oneliners also benefit from terminology normalization. A new integration test, `transcribe_worker_normalizes_whisper_output`, has been added to verify that the Whisper output is correctly normalized by the Gazetteer before being saved to the transcript file.
This commit is contained in:
+7
-3
@@ -65,8 +65,10 @@ async fn main() {
|
||||
.build()
|
||||
.expect("reqwest client build failed");
|
||||
|
||||
// Gazetteer for pre-LLM proper-name correction. Missing / unreadable
|
||||
// vocab directory is non-fatal — analysis falls back to plain prompts.
|
||||
// Gazetteer: post-AI terminology normalization filter applied to
|
||||
// every AI output (Whisper, oneliner, analyze). Missing / unreadable
|
||||
// vocab directory is non-fatal — the pipeline runs without
|
||||
// normalization.
|
||||
let vocab = Arc::new(match Gazetteer::load_dir(&config.vocab_dir).await {
|
||||
Ok(g) => {
|
||||
info!(
|
||||
@@ -95,17 +97,19 @@ async fn main() {
|
||||
config.clone(),
|
||||
http_client.clone(),
|
||||
transcribe_busy.clone(),
|
||||
vocab.clone(),
|
||||
));
|
||||
{
|
||||
let tx = transcribe_tx.clone();
|
||||
let data_path = config.data_path.clone();
|
||||
let client = http_client.clone();
|
||||
let cfg = config.clone();
|
||||
let vocab = vocab.clone();
|
||||
tokio::spawn(async move {
|
||||
transcribe::recovery::scan_and_enqueue(&data_path, &tx).await;
|
||||
// Then catch up oneliners for cases that crashed between
|
||||
// transcript-write and oneliner-write in a previous run.
|
||||
transcribe::recovery::regenerate_missing_oneliners(&data_path, &client, &cfg).await;
|
||||
transcribe::recovery::regenerate_missing_oneliners(&data_path, &client, &cfg, &vocab).await;
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user