Struct tokenizers::models::bpe::BpeBuilder

pub struct BpeBuilder { /* private fields */ }

A BpeBuilder can be used to create a BPE model with a custom configuration.

Implementations

impl BpeBuilder

pub fn new() -> Self

Constructs a new BpeBuilder.

pub fn files(self, vocab: String, merges: String) -> Self

Set the input files.

pub fn vocab_and_merges(self, vocab: Vocab, merges: Merges) -> Self

Set the vocab (token -> ID) and merges mappings.

pub fn cache_capacity(self, capacity: usize) -> Self

Set the cache’s capacity. Set to 0 if you want to disable caching.

pub fn dropout(self, dropout: f32) -> Self

Use dropout with the model.

pub fn unk_token(self, unk_token: String) -> Self

Set the UNK token for the vocab.

pub fn continuing_subword_prefix(self, prefix: String) -> Self

Set the continuing_subword_prefix option.

pub fn end_of_word_suffix(self, prefix: String) -> Self

Set the end_of_word_suffix option.

pub fn fuse_unk(self, fuse_unk: bool) -> Self

Set the fuse_unk option.

pub fn byte_fallback(self, byte_fallback: bool) -> Self

Set the byte_fallback option.

pub fn build(self) -> Result<BPE>

Returns a BPE model that uses the BpeBuilder’s configuration.

Trait Implementations

impl Default for BpeBuilder

fn default() -> Self

Returns the “default value” for a type.