Trait tokenizers::tokenizer::Decoder

pub trait Decoder {
    // Required method
    fn decode_chain(&self, tokens: Vec<String>) -> Result<Vec<String>>;

    // Provided method
    fn decode(&self, tokens: Vec<String>) -> Result<String> { ... }
}

A Decoder changes the raw tokens into its more readable form.

Required Methods

fn decode_chain(&self, tokens: Vec<String>) -> Result<Vec<String>>

Provided Methods

fn decode(&self, tokens: Vec<String>) -> Result<String>

Implementors

impl Decoder for DecoderWrapper

impl Decoder for BPEDecoder

impl Decoder for ByteFallback

impl Decoder for CTC

impl Decoder for Fuse

impl Decoder for Sequence

impl Decoder for Strip

impl Decoder for WordPiece

impl Decoder for Replace

impl Decoder for ByteLevel

As a Decoder, ByteLevel is in charge of converting any byte-level characters to their unicode counterpart, before merging everything back into a single String. This decoder will consume the tokens and merge them in one step to alleviate the fact that single token decoded might be a byte not representable as as String.

impl Decoder for Metaspace