Struct tokenizers::tokenizer::pre_tokenizer::Split

pub struct Split { /* private fields */ }

Wrapper for a subpart of a NormalizedString.

This Split contains the underlying NormalizedString as well as its offsets in the original string. These offsets are in the original referential. It also contains any Token associated to the current split

Trait Implementations

impl Clone for Split

fn clone(&self) -> Split

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Split

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<(NormalizedString, Option<Vec<Token>>)> for Split

fn from(f: (NormalizedString, Option<Vec<Token>>)) -> Self

Converts to this type from the input type.

impl From<NormalizedString> for Split

fn from(n: NormalizedString) -> Self

Converts to this type from the input type.

impl PartialEq for Split

fn eq(&self, other: &Split) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for Split

impl StructuralPartialEq for Split