Add gazetteer configuration and dependency
This commit introduces the `VOCAB_DIR` configuration option, which specifies the directory for gazetteer files. It also adds the `strsim` crate, which is used for calculating string similarity and is essential for the gazetteer's functionality. The gazetteer module has been significantly expanded to include the `Gazetteer` struct, its loading mechanism from `.txt` files, and an annotation function. This functionality allows for pre-LLM correction of proper names and technical vocabulary by identifying potential misspellings based on phonetic similarity and edit distance.
This commit is contained in:
Generated
+7
@@ -345,6 +345,7 @@ dependencies = [
|
||||
"rpassword",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"strsim",
|
||||
"tempfile",
|
||||
"time",
|
||||
"tokio",
|
||||
@@ -1642,6 +1643,12 @@ version = "1.2.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596"
|
||||
|
||||
[[package]]
|
||||
name = "strsim"
|
||||
version = "0.11.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f"
|
||||
|
||||
[[package]]
|
||||
name = "subtle"
|
||||
version = "2.6.1"
|
||||
|
||||
Reference in New Issue
Block a user