Enum tokenizers::tokenizer::normalizer::SplitDelimiterBehavior

pub enum SplitDelimiterBehavior {
    Removed,
    Isolated,
    MergedWithPrevious,
    MergedWithNext,
    Contiguous,
}

Defines the expected behavior for the delimiter of a Split Pattern When splitting on '-' for example, with input the-final--countdown:

• Removed => [ "the", "final", "countdown" ]
• Isolated => [ "the", "-", "final", "-", "-", "countdown" ]
• MergedWithPrevious => [ "the-", "final-", "-", "countdown" ]
• MergedWithNext => [ "the", "-final", "-", "-countdown" ]
• Contiguous => [ "the", "-", "final", "--", "countdown" ]

Variants

Removed

Isolated

MergedWithPrevious

MergedWithNext

Contiguous

Trait Implementations

impl Clone for SplitDelimiterBehavior

fn clone(&self) -> SplitDelimiterBehavior

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for SplitDelimiterBehavior

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for SplitDelimiterBehavior

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq for SplitDelimiterBehavior

fn eq(&self, other: &SplitDelimiterBehavior) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for SplitDelimiterBehavior

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl Copy for SplitDelimiterBehavior

impl Eq for SplitDelimiterBehavior

impl StructuralPartialEq for SplitDelimiterBehavior