Rate limits

To ensure a stable and reliable experience for all users, our services include rate limits on API requests to maintain fair resource distribution. This guide explains our rate limit policy and provides best practices to help you avoid hitting limits and manage exceeded requests effectively.

What are rate limits?

Rate limits help ensure a smooth and reliable experience by preventing excessive API requests from impacting system performance. They define the maximum number of API calls allowed within a specific time frame. If you exceed this limit, additional requests will be temporarily blocked until the cooldown period ends, allowing you to resume usage. The following rate limit policies are applied in our services:

• RPS (Requests per second): The maximum number of API requests allowed per second.
• RPM (Requests per minute): The maximum number of API requests allowed per minute.
• TPM (Tokens per minute): The maximum number of tokens allowed per minute (applied to text-based models like Solar LLMs).
• PPM (Pages per minute): The maximum number of pages allowed per minute (applied to document processing models like Document Parse).

What happens when the rate limit is exceeded?

If the rate limit is exceeded, the server responds with an HTTP status code 429 Too Many Requests. When this happens, you must wait for a designated cooldown period before retrying your request.

Rate limits by capability

The following table provides a comprehensive overview of rate limits across all our capabilities, organized by commitment tier:

Rate limits are automatically assigned based on your commitment tier. For detailed information about commitment tiers and pricing, please visit our Pricing page. To raise your rate limits, purchase credits on the Credit page to upgrade your commitment tier. If no commitment tier covers the limits you need, contact us with your requirements.

Generate (Chat / Reasoning / Structured outputs)

Decide (Structured decisions)

Solar Decide is served on Solar Mini 4 and shares its rate limits.

Embed

Parse (Document parsing / Document OCR)

Extract (Universal information extraction)

Classify (Document classification)

Rate limit response headers

Upstage API includes HTTP response headers in all responses to indicate the rate limit status. You can use these headers to programmatically monitor your current usage and remaining allowance, helping you proactively prevent exceeding the rate limits.

Header categories

Depending on the characteristics of the API endpoint, up to three categories of rate limit headers are included in the response.

• The RPS/RPM headers are always included.
• The TPM headers are included only if a TPM limit is set for the model.
• The PPM headers are included only if a PPM limit is set for the model.

Complete list of headers

Limit (Maximum allowance)

Indicates the maximum allowance within the current rate limit window.

Remaining (Remaining allowance)

Indicates the remaining allowance within the current rate limit window.

Reset (Reset time)

Indicates the time when the rate limit window resets as a Unix timestamp (in seconds).

Interval (Window period)

Indicates the duration unit of the rate limit window.

Retry-After (Retry possible time)

Included only when the rate limit is exceeded (HTTP status code 429). It indicates the time when you can retry the request as a Unix timestamp (in seconds).

Response examples

Successful response (Chat API — RPM + TPM applied)

HTTP/1.1 200 OK
X-Upstage-RateLimit-Limit-Requests: 500
X-Upstage-RateLimit-Remaining-Requests: 499
X-Upstage-RateLimit-Reset-Requests: 1742300460
X-Upstage-RateLimit-Interval-Requests: minute
X-Upstage-RateLimit-Limit-Tokens: 200000
X-Upstage-RateLimit-Remaining-Tokens: 195000
X-Upstage-RateLimit-Reset-Tokens: 1742300460
X-Upstage-RateLimit-Interval-Tokens: minute

Successful response (Document Parse API — RPM + PPM applied)

HTTP/1.1 200 OK
X-Upstage-RateLimit-Limit-Requests: 100
X-Upstage-RateLimit-Remaining-Requests: 98
X-Upstage-RateLimit-Reset-Requests: 1742300460
X-Upstage-RateLimit-Interval-Requests: minute
X-Upstage-RateLimit-Limit-Pages: 500
X-Upstage-RateLimit-Remaining-Pages: 490
X-Upstage-RateLimit-Reset-Pages: 1742300460
X-Upstage-RateLimit-Interval-Pages: minute

Rate limit exceeded response (HTTP 429 — RPM exceeded)

HTTP/1.1 429 Too Many Requests
X-Upstage-RateLimit-Limit-Requests: 500
X-Upstage-RateLimit-Remaining-Requests: 0
X-Upstage-RateLimit-Reset-Requests: 1742300460
X-Upstage-RateLimit-Interval-Requests: minute
X-Upstage-RateLimit-Retry-After-Requests: 1742300445
X-Upstage-RateLimit-Limit-Tokens: 200000
X-Upstage-RateLimit-Remaining-Tokens: 150000
X-Upstage-RateLimit-Reset-Tokens: 1742300460
X-Upstage-RateLimit-Interval-Tokens: minute

Programming examples

Python

import time
import requests

response = requests.post(
    "https://api.upstage.ai/v1/solar/chat/completions",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "model": "solar-pro",
        "messages": [{"role": "user", "content": "Hello"}]
    }
)

# Check rate limit status
rpm_remaining = int(response.headers.get("X-Upstage-RateLimit-Remaining-Requests", -1))
rpm_reset = int(response.headers.get("X-Upstage-RateLimit-Reset-Requests", 0))

if rpm_remaining == 0:
    # Wait until rate limit resets
    wait_seconds = rpm_reset - int(time.time())
    if wait_seconds > 0:
        print(f"Rate limit reached. Waiting {wait_seconds}s...")
        time.sleep(wait_seconds)

# Handle 429 response
if response.status_code == 429:
    retry_after = int(response.headers.get("X-Upstage-RateLimit-Retry-After-Requests", 0))
    wait_seconds = retry_after - int(time.time())
    if wait_seconds > 0:
        time.sleep(wait_seconds)

cURL

# Check only response headers
curl -s -D - -o /dev/null \
  -H "Authorization: Bearer $UPSTAGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "solar-pro", "messages": [{"role": "user", "content": "Hello"}]}' \
  https://api.upstage.ai/v1/solar/chat/completions \
  | grep -i "X-Upstage-RateLimit"

FAQ

Q: Are all 15 headers included in every API response?

No. Only the rate limit headers configured for the specific model are included by category.

• Chat, Embedding models: RPM + TPM headers (up to 10 headers)
• Document Parse, OCR models: RPM + PPM headers (up to 10 headers)
• RPM headers are always included in all API requests.

Q: What is the difference between Reset and Retry-After?

• Reset: The time when the rate limit window completely resets. After this time, the entire allowance is fully restored.
• Retry-After: Included only when the rate limit is exceeded (429), indicating the earliest time you can retry the request. It is earlier than or equal to the Reset time.

Q: Can the Interval value be anything other than minute?

In the code, two values are possible: second and minute. Currently, most rate limits are set in minute units, but this may change in the future.

Q: Can header values be negative?

No. The Remaining value does not go below 0. When you reach the limit, it shows 0, and the X-Upstage-RateLimit-Retry-After-* header tells you when you can retry.

Best practices to handle rate limits

Implement retry logic with backoff

Implement a retry logic that pauses API calls for a certain period before attempting another request. This ensures that your requests are spaced out enough to avoid hitting the rate limit repeatedly. For instance, if the limit is per minute, you can set a delay before the next request or use exponential backoff, where the wait time increases progressively after each retry.

import time

import openai
from openai import OpenAI  # openai==1.52.2

documents = [
    "Solar embeddings are awesome",
    "Sam is a teacher.",
    "This is another document",
    "Hi, how are you?",
    "Father went into his room",
    "Mauna Kea is an inactive volcano on the island of Hawaiʻi. Its peak is 4,207.3 m above sea level, making it the highest point in Hawaii and second-highest peak of an island on Earth.",
    ...
]

client = OpenAI(
    api_key="UPSTAGE_API_KEY_INJECT",
    base_url="https://api.upstage.ai/v1"
)

embeddings = []

limit_attempts = 2

for i in range(0, len(documents)):
    num_attempts = 0
    wait_time = 1
    while num_attempts < limit_attempts:
        num_attempts += 1

        try:
            response = client.embeddings.create(
                input=documents[i],
                model="embedding-query"
            )

            embeddings += response.data[0].embedding
            print(f"Successfully processed document {i+1} of {len(documents)}")
            print(f"Embedding: {embeddings[i]}")
        except openai.RateLimitError:
            time.sleep(wait_time)
            wait_time *= 2

Use asynchronous APIs

Asynchronous APIs typically support higher rate limits, making them a more efficient option for handling large volumes of requests. Asynchronous APIs are available for some of our models such as Document Parse. This approach helps distribute API calls over time, reducing the risk of exceeding rate limits while optimizing performance.

Batch processing

Instead of sending frequent individual API requests, consider using batch processing. By grouping multiple requests into a single batch, you can reduce the number of API calls and stay within the rate limit. This is particularly useful for large-scale embeddings operations where multiple requests need to be sent in a short time.

from openai import OpenAI  # openai==1.52.2

documents = [
    "Solar embeddings are awesome",
    "Sam is a teacher.",
    "This is another document",
    "Hi, how are you?",
    "Father went into his room",
    "Mauna Kea is an inactive volcano on the island of Hawaiʻi. Its peak is 4,207.3 m above sea level, making it the highest point in Hawaii and second-highest peak of an island on Earth.",
    ...
]

client = OpenAI(
    api_key="UPSTAGE_API_KEY_INJECT",
    base_url="https://api.upstage.ai/v1"
)

batch_size = 32
embeddings = []

for i in range(0, len(documents), batch_size):
    response = client.embeddings.create(
        input=documents[i:i + batch_size],
        model="embedding-query"
    )

    for datum in response.data:
        embeddings += datum.embedding
        print(f"Successfully processed document {i+1} of {len(documents)}")
        print(f"Embedding: {embeddings[i]}")

Caching strategy

If your API requests frequently return the same data, implementing a caching strategy can greatly reduce the number of requests. This is particularly beneficial in scenarios like embeddings.

Store the results of frequent requests and serve them from the cache instead of making repeated API calls. This reduces the load on the API and helps in staying within the allowed limits.

from openai import OpenAI  # openai==1.52.2

documents = [
    "Solar embeddings are awesome",
    "Sam is a teacher.",
    "This is another document",
    "Hi, how are you?",
    "Father went into his room",
    "Mauna Kea is an inactive volcano on the island of Hawaiʻi. Its peak is 4,207.3 m above sea level, making it the highest point in Hawaii and second-highest peak of an island on Earth.",
    ...
]

client = OpenAI(
    api_key="UPSTAGE_API_KEY_INJECT",
    base_url="https://api.upstage.ai/v1"
)

cache = {}

embeddings = []

for i in range(0, len(documents)):
    document = documents[i]

    if document not in cache:
        response = client.embeddings.create(
            input=document,
            model="embedding-query"
        )
        cache[document] = response.data[0].embedding

    embeddings += cache[document]
    print(f"Successfully processed document {i+1} of {len(documents)}")
    print(f"Embedding: {embeddings[i]}")

Still need a higher rate limit?

Rate limits are provided based on your commitment tier. Higher tiers unlock increased rate limits and additional benefits.

If you need higher limits right away, you can upgrade your tier by purchasing credits:

• One-time purchase: Benefits last for one month from the time the tier is activated.
• Monthly/Annual auto-recharge: Benefits renew with each billing cycle, keeping your tier active continuously.

For custom enterprise agreements, please contact us with your requirements and we'll have a business representative reach out to you.