Enum tokenizers::models::bpe::Error

pub enum Error {
    Io(Error),
    JsonError(Error),
    BadVocabulary,
    BadMerges(usize),
    MergeTokenOutOfVocabulary(String),
    UnkTokenOutOfVocabulary(String),
    InvalidDropout,
}

Errors that can be encountered while using or constructing a BPE model.

Variants

Io(Error)

An error encountered while reading files mainly.

JsonError(Error)

An error forwarded from Serde, while parsing JSON

BadVocabulary

When the vocab.json file is in the wrong format

BadMerges(usize)

When the merges.txt file is in the wrong format. This error holds the line number of the line that caused the error.

MergeTokenOutOfVocabulary(String)

If a token found in merges, is not in the vocab

UnkTokenOutOfVocabulary(String)

If the provided unk token is out of vocabulary

InvalidDropout

Dropout not between 0 and 1.

Trait Implementations

impl Debug for Error

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for Error

fn fmt(&self, __formatter: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Error for Error

fn source(&self) -> Option<&(dyn Error + 'static)>

The lower-level source of this error, if any.

fn description(&self) -> &str

👎Deprecated since 1.42.0: use the Display impl or to_string()

fn cause(&self) -> Option<&dyn Error>

👎Deprecated since 1.33.0: replaced by Error::source, which can support downcasting

fn provide<'a>(&'a self, request: &mut Request<'a>)

🔬This is a nightly-only experimental API. (error_generic_member_access)

Provides type based access to context intended for error reports.

impl From<Error> for Error

fn from(source: Error) -> Self

Converts to this type from the input type.

impl From<Error> for Error

fn from(source: Error) -> Self

Converts to this type from the input type.