Counting tokens

On this page, you will learn what tokens and tokenizers are, and how to count tokens in your requests to Solar models.

What is token and tokenizer?

A token is a piece of text, such as a word, subword, or even character, that the model processes individually. Tokenization is the process of breaking down text into tokens. A tokenizer is a tool or algorithm that performs this task. It converts raw text into a sequence of tokens, which the model can then use for training or inference. The tokenizer ensures that the input text is broken down in a way that balances efficiency and accuracy, enabling the LLM to generate meaningful and coherent responses.

Context length of the model

Your request to the Solar model is encoded into tokens and fed to the model to generate output. Solar models have limitations on maximum number of tokens they can handle at a time, known as context length. Refer to the context length columns on Chat page to check these limitations.

For text generation models (Chat, Translation, Groundedness Check), the context length includes both input and output tokens. For example, if model’s context length is 4096 and the input length is 3096, it can generate up to 1000 tokens. Therefore, to generate the desired output, you need to request proper amount of tokens to model at a time. If the sum of input tokens and max_tokens exceeds context length of model, You will receive a 400 status code.

For embedding models, the context length only includes input tokens. The number of input tokens should be less than or equal to the model's context length.

Depending on the model and your request, additional tokens might be appended to your request before fed to model. Check out this notebook for more information.

How to count tokens

You can use the HuggingFace Tokenizers python library to count the tokens in your request.

Install Library

pip install tokenizers==0.20.0

Count tokens and truncate tokens

from tokenizers import Tokenizer

# Check out the tokenizers listed below.
# If you're using an Embeddings model, make sure to use "upstage/solar-1-mini-tokenizer".
tokenizer = Tokenizer.from_pretrained("upstage/solar-pro4-tokenizer")

text = "Hi, how are you?"
enc = tokenizer.encode(text)
print("Encoded input:")
print(enc)

inv_vocab = {v: k for k, v in tokenizer.get_vocab().items()}
tokens = [inv_vocab[token_id] for token_id in enc.ids]
print("Tokens:")
print(tokens)

number_of_tokens = len(enc.ids)
print("Number of tokens:", number_of_tokens)

# Truncate tokens
token_length_limit = 4000
truncated_text = tokenizer.decode(enc.ids[:4000])
print("Truncated text: ")
print(truncated_text)

Here is a list of tokenizers and their respective supported models.