Module tokenizers::models::bpe

Byte Pair Encoding model.

Re-exports

• pub use trainer::*;

Modules

• trainer

Structs

• BPE
  A Byte Pair Encoding model.
• BpeBuilder
  A BpeBuilder can be used to create a BPE model with a custom configuration.

Enums

• Error
  Errors that can be encountered while using or constructing a BPE model.

Type Aliases

• MergeMap
• Merges
• Vocab