Add gazetteer to analyze worker
Pass a `Gazetteer` to the analyze worker to enable proper-name correction. The gazetteer is loaded from disk at application startup. If the directory is missing or unreadable, the worker will start without proper-name correction capabilities.
This commit is contained in:
@@ -7,18 +7,26 @@ use tracing::{error, info, warn};
|
||||
|
||||
use super::{llm, prompt, AnalysisInput, AnalyzeReceiver, ANALYSIS_INPUT_FILE, DOCUMENT_FILE};
|
||||
use crate::config::Config;
|
||||
use crate::gazetteer::Gazetteer;
|
||||
use crate::{BusyGuard, WorkerBusy};
|
||||
|
||||
/// Consume analyze jobs sequentially. The external LLM tolerates parallelism,
|
||||
/// but sequential processing keeps the architecture simple and shields the
|
||||
/// provider from bursts. Trivial to lift later via `tokio::spawn(process(..))`.
|
||||
///
|
||||
/// `_vocab` is the gazetteer for proper-name correction. Currently unused;
|
||||
/// wired into `process()` in a follow-up step.
|
||||
pub async fn run(
|
||||
mut rx: AnalyzeReceiver,
|
||||
config: Arc<Config>,
|
||||
client: reqwest::Client,
|
||||
worker_busy: WorkerBusy,
|
||||
_vocab: Arc<Gazetteer>,
|
||||
) {
|
||||
info!("Analyze worker started");
|
||||
info!(
|
||||
vocab_entries = _vocab.len(),
|
||||
"Analyze worker started"
|
||||
);
|
||||
let timeout = Duration::from_secs(config.llm_timeout_seconds);
|
||||
|
||||
while let Some(job) = rx.recv().await {
|
||||
|
||||
Reference in New Issue
Block a user