Trait tokenizers::tokenizer::Normalizer

source · 
pub trait Normalizer {
    // Required method
    fn normalize(&self, normalized: &mut NormalizedString) -> Result<()>;
}
Expand description

Takes care of pre-processing strings.

Required Methods§

source

fn normalize(&self, normalized: &mut NormalizedString) -> Result<()>

Implementors§