Struct grpc_vision_svc::image_captioning::token_output_stream::TokenOutputStream

pub struct TokenOutputStream { /* private fields */ }

TokenOutputStream is a wrapper around a tokenizer that allows for streaming tokens to the user rather than waiting for the full decoding to complete.

Implementations

impl TokenOutputStream

pub fn new(tokenizer: Tokenizer) -> Self

Creates a new TokenOutputStream instance.

Arguments

• tokenizer - A Tokenizer instance to be used for tokenizing.

Returns

A new TokenOutputStream instance.

pub fn into_inner(self) -> Tokenizer

Consumes the TokenOutputStream, returning the inner Tokenizer.

This method is used when the TokenOutputStream is no longer needed, and you want to access the underlying Tokenizer.

Returns

The inner Tokenizer instance.

pub fn tokenizer(&self) -> &Tokenizer

Provides a reference to the inner Tokenizer.

This method is used when you want to access the underlying Tokenizer but still keep the TokenOutputStream for further use.

Returns

A reference to the inner Tokenizer instance.

pub fn get_token(&self, token_s: &str) -> Option<u32>

Retrieves the token associated with a given string.

Arguments

• token_s - A string representation of the token.

Returns

An Option which contains the token if it exists, or None if it does not.

pub fn clear(&mut self)

Clears the TokenOutputStream.

This method is used to reset the state of the TokenOutputStream. It clears the tokens and resets the prev_index and current_index to 0.

pub fn next_token(&mut self, token: u32) -> Result<Option<String>>

Processes the next token and returns the decoded string if the token leads to a new word.

Arguments

• token - The next token to process.

Returns

A Result which contains an Option with the decoded string if the token leads to a new word, or None if it does not. Returns an error if the decoding fails.

Example

// Assuming that the `tokenizer.json` file contains the following vocab:
// { "hello": 1, "world": 2, "everybody": 3 }
let tokenizer = Tokenizer::from_file("path/to/tokenizer.json").unwrap();
let mut stream = TokenOutputStream::new(tokenizer);

let tokens: [u32; 4] = [1, 2, 1, 3];

let sent: String = tokens
    .iter()
    .filter_map(|token| stream.next_token(*token).ok())
    .flatten()
    .collect();

assert_eq!(sent, "hello world hello everybody");

pub fn decode_rest(&self) -> Result<Option<String>>

Decodes the remaining tokens and returns the decoded string if there are any new words.

Returns

A Result which contains an Option with the decoded string if there are any new words, or None if there are not. Returns an error if the decoding fails.

pub fn decode_all(&self) -> Result<String>

Decodes all tokens in the TokenOutputStream and returns the decoded string.

Returns

A Result which contains the decoded string if the decoding is successful, or an error if the decoding fails.

Trait Implementations

impl Debug for TokenOutputStream

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.