Trait tokenizers::utils::parallelism::MaybeParallelIterator

pub trait MaybeParallelIterator<P, S>
where
    P: ParallelIterator,
    S: Iterator<Item = P::Item>,
{
    // Required methods
    fn into_maybe_par_iter(self) -> CondIterator<P, S>;
    fn into_maybe_par_iter_cond(self, cond: bool) -> CondIterator<P, S>;
}

Allows to convert into an iterator that can be executed either parallelly or serially.

The choice is made according to the currently set TOKENIZERS_PARALLELISM environment variable. This variable can have one of the following values

• False => “” (empty value), “false”, “f”, “off”, “no”, “n”, “0”
• True => Any other value

Required Methods

fn into_maybe_par_iter(self) -> CondIterator<P, S>

Convert ourself in a CondIterator, that will be executed either in parallel or serially, based solely on the TOKENIZERS_PARALLELISM environment variable

fn into_maybe_par_iter_cond(self, cond: bool) -> CondIterator<P, S>

Convert ourself in a CondIterator, that will be executed either in parallel or serially, based on both the TOKENIZERS_PARALLELISM environment variable and the provided bool. Both must be true to run with parallelism activated.

Implementors

impl<P, S, I> MaybeParallelIterator<P, S> for I

where I: IntoParallelIterator<Iter = P, Item = P::Item> + IntoIterator<IntoIter = S, Item = S::Item>, P: ParallelIterator, S: Iterator<Item = P::Item>,