Module tokenizers::pre_tokenizers::digits

source ·

Structs§

  • Digits
    Pre tokenizes the numbers into single tokens. If individual_digits is set to true, then all digits are splitted into individual tokens.