Module tokenizers::models::unigram

source ·
Expand description

Unigram model.

Structs§

  • Lattice
    Structure to implement Viterbi algorithm to find the best encoding, or sample from all possible encodings of a given sentence.
  • Node
    A node from the lattice, that helps reconstruct the underlying String
  • Unigram
    A Unigram model to encode sentences.
  • UnigramIterator
    Iterator to iterate of vocabulary of the model, and their relative score.
  • UnigramTrainer
    A UnigramTrainer can train a Unigram model from word_counts.
  • UnigramTrainerBuilder
    Builder for UnigramTrainer.

Enums§