Struct tokenizers::models::unigram::UnigramTrainer
source · #[non_exhaustive]pub struct UnigramTrainer {
pub show_progress: bool,
pub vocab_size: u32,
pub n_sub_iterations: u32,
pub shrinking_factor: f64,
pub special_tokens: Vec<AddedToken>,
pub initial_alphabet: HashSet<char>,
pub unk_token: Option<String>,
pub max_piece_length: usize,
/* private fields */
}
Expand description
A UnigramTrainer
can train a Unigram
model from word_counts
.
Fields (Non-exhaustive)§
This struct is marked as non-exhaustive
Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional
Struct { .. }
syntax; cannot be matched against without a wildcard ..
; and struct update syntax will not work.show_progress: bool
§vocab_size: u32
§n_sub_iterations: u32
§shrinking_factor: f64
§special_tokens: Vec<AddedToken>
§initial_alphabet: HashSet<char>
§unk_token: Option<String>
§max_piece_length: usize
Implementations§
Trait Implementations§
source§impl Clone for UnigramTrainer
impl Clone for UnigramTrainer
source§fn clone(&self) -> UnigramTrainer
fn clone(&self) -> UnigramTrainer
Returns a copy of the value. Read more
1.0.0 · source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
Performs copy-assignment from
source
. Read moresource§impl Debug for UnigramTrainer
impl Debug for UnigramTrainer
source§impl Default for UnigramTrainer
impl Default for UnigramTrainer
source§impl<'de> Deserialize<'de> for UnigramTrainer
impl<'de> Deserialize<'de> for UnigramTrainer
source§fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
Deserialize this value from the given Serde deserializer. Read more
source§impl From<UnigramTrainer> for TrainerWrapper
impl From<UnigramTrainer> for TrainerWrapper
source§fn from(from: UnigramTrainer) -> Self
fn from(from: UnigramTrainer) -> Self
Converts to this type from the input type.
source§impl Serialize for UnigramTrainer
impl Serialize for UnigramTrainer
Auto Trait Implementations§
impl Freeze for UnigramTrainer
impl RefUnwindSafe for UnigramTrainer
impl Send for UnigramTrainer
impl Sync for UnigramTrainer
impl Unpin for UnigramTrainer
impl UnwindSafe for UnigramTrainer
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more