Struct tokenizers::tokenizer::pre_tokenizer::PreTokenizedString

pub struct PreTokenizedString { /* private fields */ }

The PreTokenizedString is in charge of splitting an underlying string, making sure everything is fine while doing so, and providing ways to normalize and tokenize these splits. Once everything has been normalized and tokenized, the PreTokenizedString is able to build an Encoding with all the relevant offsets and word ids, relative to the original string.

Implementations

impl PreTokenizedString

pub fn split<F, U, R>(&mut self, split_fn: F) -> Result<()>

where F: FnMut(usize, NormalizedString) -> Result<U>, U: IntoIterator<Item = R>, R: Into<Split>,

Split the PreTokenizedString by providing a split_fn in charge of splitting each substring (NormalizedString) into multiple parts.

split_fn takes a NormalizedString and is in charge of returning an iterator over the produced NormalizedString. split_fn is free of modifying these NormalizedString as relevant, as long as it respects the constraint stated below.

There are only one constraint that MUST be respected:

The produced NormalizedString, if combined back together, must have the same original string as the original one given to split_fn. This concretely means that for the offset tracking to work as expected, split_fn must produce “splits” of the original string.

pub fn normalize<F>(&mut self, normalize: F) -> Result<()>

where F: Fn(&mut NormalizedString) -> Result<()>,

Normalized all the splits that do not have attached Tokens, using the provided normalize function.

pub fn tokenize<F>(&mut self, tokenize: F) -> Result<()>

where F: Fn(&NormalizedString) -> Result<Vec<Token>>,

Tokenize all the splits that do not have attached Tokens, using the provided tokenize function

pub fn into_encoding( self, word_idx: Option<u32>, type_id: u32, offset_type: OffsetType ) -> Result<Encoding>

Transform the current PreTokenizedString into an Encoding.

If a word_idx is provided, any word in the generated Encoding will be set to this value. This is generally used with pre-tokenized input, that do not need the PreTokenizedString to generate word ids.

This method will fail if some splits do not have associated Token.

pub fn get_splits( &self, offset_ref: OffsetReferential, offset_type: OffsetType ) -> Vec<(&str, Offsets, &Option<Vec<Token>>)>

Returns a list of splits, each of them being a slice of the normalized string, the associated offsets either in original or normalized referential, as well as the potention tokens

Trait Implementations

impl Clone for PreTokenizedString

fn clone(&self) -> PreTokenizedString

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PreTokenizedString

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<&str> for PreTokenizedString

fn from(s: &str) -> Self

Converts to this type from the input type.

impl From<NormalizedString> for PreTokenizedString

fn from(s: NormalizedString) -> Self

Converts to this type from the input type.

impl From<String> for PreTokenizedString

fn from(s: String) -> Self

Converts to this type from the input type.

impl PartialEq for PreTokenizedString

fn eq(&self, other: &PreTokenizedString) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for PreTokenizedString

impl StructuralPartialEq for PreTokenizedString