Struct tokenizers::processors::template::Tokens

pub struct Tokens(pub HashMap<String, SpecialToken>);

A bunch of SpecialToken represented by their ID. Internally, Tokens is a HashMap<String, SpecialToken> and can be built from a HashMap or a Vec<SpecialToken>.

Tuple Fields

0: HashMap<String, SpecialToken>

Trait Implementations

impl Clone for Tokens

fn clone(&self) -> Tokens

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Tokens

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Tokens

fn default() -> Tokens

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for Tokens

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl From<HashMap<String, SpecialToken>> for Tokens

fn from(v: HashMap<String, SpecialToken>) -> Self

Converts to this type from the input type.

impl<T: Into<SpecialToken>> From<Vec<T>> for Tokens

fn from(v: Vec<T>) -> Self

Converts to this type from the input type.

impl PartialEq for Tokens

fn eq(&self, other: &Tokens) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for Tokens

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl Eq for Tokens

impl StructuralPartialEq for Tokens