Module tokenizers::decoders::bpe

source ·

Structs§

  • BPEDecoder
    Allows decoding Original BPE by joining all the tokens and then replacing the suffix used to identify end-of-words by whitespaces