Module tokenizers::models

Popular tokenizer models.

Modules

• bpe
  Byte Pair Encoding model.
• unigram
  Unigram model.
• wordlevel
• wordpiece
  WordPiece model.

Enums

• ModelWrapper
• TrainerWrapper