Module tokenizers::pre_tokenizers::byte_level

Structs

• ByteLevel
  Provides all the necessary steps to handle the BPE tokenization at the byte-level. Takes care of all the required processing steps to transform a UTF-8 string as needed before and after the BPE model does its job.

Functions

• process_offsets