Struct tokenizers::normalizers::bert::BertNormalizer
source · #[non_exhaustive]pub struct BertNormalizer {
pub clean_text: bool,
pub handle_chinese_chars: bool,
pub strip_accents: Option<bool>,
pub lowercase: bool,
}
Fields (Non-exhaustive)§
This struct is marked as non-exhaustive
Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional
Struct { .. }
syntax; cannot be matched against without a wildcard ..
; and struct update syntax will not work.clean_text: bool
Whether to do the bert basic cleaning:
- Remove any control characters
- Replace all sorts of whitespace by the classic one
handle_chinese_chars: bool
Whether to put spaces around chinese characters so they get split
strip_accents: Option<bool>
Whether to strip accents
lowercase: bool
Whether to lowercase the input
Implementations§
Trait Implementations§
source§impl Clone for BertNormalizer
impl Clone for BertNormalizer
source§fn clone(&self) -> BertNormalizer
fn clone(&self) -> BertNormalizer
Returns a copy of the value. Read more
1.0.0 · source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
Performs copy-assignment from
source
. Read moresource§impl Debug for BertNormalizer
impl Debug for BertNormalizer
source§impl Default for BertNormalizer
impl Default for BertNormalizer
source§impl<'de> Deserialize<'de> for BertNormalizer
impl<'de> Deserialize<'de> for BertNormalizer
source§fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
Deserialize this value from the given Serde deserializer. Read more
source§impl From<BertNormalizer> for NormalizerWrapper
impl From<BertNormalizer> for NormalizerWrapper
source§fn from(from: BertNormalizer) -> Self
fn from(from: BertNormalizer) -> Self
Converts to this type from the input type.
source§impl Normalizer for BertNormalizer
impl Normalizer for BertNormalizer
source§impl Serialize for BertNormalizer
impl Serialize for BertNormalizer
impl Copy for BertNormalizer
Auto Trait Implementations§
impl Freeze for BertNormalizer
impl RefUnwindSafe for BertNormalizer
impl Send for BertNormalizer
impl Sync for BertNormalizer
impl Unpin for BertNormalizer
impl UnwindSafe for BertNormalizer
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more