Enum tokenizers::tokenizer::normalizer::Range

pub enum Range<T: RangeBounds<usize> + Clone> {
    Original(T),
    Normalized(T),
}

Represents a Range usable by the NormalizedString to index its content. A Range can use indices relative to either the Original or the Normalized string

Variants

Original(T)

Normalized(T)

Implementations

impl<T> Range<T>

where T: RangeBounds<usize> + Clone,

pub fn unwrap(self) -> T

Unwrap the underlying range

pub fn len(&self) -> Option<usize>

Return the length of the current Range if not Unbounded

pub fn into_full_range(self, max_len: usize) -> Range<usize>

Converts the current Range to a std::ops::Range<usize>. This requires the max_len of the represented string (in chars, not bytes) in order to cover the case where the original provided range was unbounded

Trait Implementations

impl<T: Clone + RangeBounds<usize> + Clone> Clone for Range<T>

fn clone(&self) -> Range<T>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<T: Debug + RangeBounds<usize> + Clone> Debug for Range<T>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<T: PartialEq + RangeBounds<usize> + Clone> PartialEq for Range<T>

fn eq(&self, other: &Range<T>) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<T: Eq + RangeBounds<usize> + Clone> Eq for Range<T>

impl<T: RangeBounds<usize> + Clone> StructuralPartialEq for Range<T>