Module tokenizers::pre_tokenizers::metaspace

source ·

Structs§

  • Metaspace
    Replaces all the whitespaces by the provided meta character and then splits on this character

Enums§

  • PrependScheme
    Enum representing options for the metaspace prepending scheme.