Struct tokenizers::normalizers::Precompiled

pub struct Precompiled { /* private fields */ }

This struct is specifically done to be compatible with SentencePiece SentencePiece models embed their Normalizer within a precompiled_charsmap that both represents a Trie, and embedded rewrite rules. In order to be 100% compliant we need to interpret that binary format too. The format is [u32 (length of trie), trie: u32, normalized: String] The trie has u8 as entries, and u32 as values, those u32 values point to offsets withing the String that correspond to the real replace value The normalized string contains ‘\0’ that should indicate the end of an entry.

Hence, normalized could be “abc\0”, some entry in the trie could be 0 meaning the value is “abc” and another one be 1 meaning the actual entry was “bc”.

Implementations

impl Precompiled

pub fn from( precompiled_charsmap: &[u8] ) -> Result<Precompiled, PrecompiledError>

pub fn transform(&self, chunk: &str) -> Option<&str>

pub fn normalize_string(&self, original: &str) -> String

Trait Implementations

impl Clone for Precompiled

fn clone(&self) -> Precompiled

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Precompiled

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for Precompiled

fn default() -> Precompiled

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for Precompiled

fn deserialize<__D>( __deserializer: __D ) -> Result<Precompiled, <__D as Deserializer<'de>>::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl From<Precompiled> for NormalizerWrapper

fn from(from: Precompiled) -> Self

Converts to this type from the input type.

impl Normalizer for Precompiled

fn normalize(&self, normalized: &mut NormalizedString) -> Result<()>

impl PartialEq for Precompiled

fn eq(&self, other: &Precompiled) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for Precompiled

fn serialize<__S>( &self, __serializer: __S ) -> Result<<__S as Serializer>::Ok, <__S as Serializer>::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for Precompiled