Add gazetteer configuration and dependency
This commit introduces the `VOCAB_DIR` configuration option, which specifies the directory for gazetteer files. It also adds the `strsim` crate, which is used for calculating string similarity and is essential for the gazetteer's functionality. The gazetteer module has been significantly expanded to include the `Gazetteer` struct, its loading mechanism from `.txt` files, and an annotation function. This functionality allows for pre-LLM correction of proper names and technical vocabulary by identifying potential misspellings based on phonetic similarity and edit distance.
This commit is contained in:
@@ -26,6 +26,7 @@ time = { version = "0.3.47", features = ["local-offset", "parsing", "formatting"
|
||||
toml_edit = "0.25.11"
|
||||
rpassword = "7.4.0"
|
||||
pulldown-cmark = { version = "0.13", default-features = false, features = ["html"] }
|
||||
strsim = "0.11"
|
||||
|
||||
[dev-dependencies]
|
||||
tower = { version = "0.5", features = ["util"] }
|
||||
|
||||
Reference in New Issue
Block a user