Struct tokenizers::normalizers::bert::BertNormalizer

#[non_exhaustive]
pub struct BertNormalizer {
    pub clean_text: bool,
    pub handle_chinese_chars: bool,
    pub strip_accents: Option<bool>,
    pub lowercase: bool,
}

Fields (Non-exhaustive)

Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.

clean_text: bool

Whether to do the bert basic cleaning:

1. Remove any control characters
2. Replace all sorts of whitespace by the classic one

handle_chinese_chars: bool

Whether to put spaces around chinese characters so they get split

strip_accents: Option<bool>

Whether to strip accents

lowercase: bool

Whether to lowercase the input

Implementations

impl BertNormalizer

pub fn new( clean_text: bool, handle_chinese_chars: bool, strip_accents: Option<bool>, lowercase: bool ) -> Self

Trait Implementations

impl Clone for BertNormalizer

fn clone(&self) -> BertNormalizer

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for BertNormalizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for BertNormalizer

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for BertNormalizer

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl From<BertNormalizer> for NormalizerWrapper

fn from(from: BertNormalizer) -> Self

Converts to this type from the input type.

impl Normalizer for BertNormalizer

fn normalize(&self, normalized: &mut NormalizedString) -> Result<()>

impl Serialize for BertNormalizer

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl Copy for BertNormalizer