Overview¶
What are rate limits?¶
A rate limit is a restriction that an API imposes on the number of times a user or client can access the server within a specified period of time.
How do I know if I am rate limited?¶
Per standard HTTP practices, your request will receive a response with HTTP status code of 429, Too Many Requests.
What are the rate limits for our API?¶
The LLM Engine API is currently in a preview mode, and therefore we currently do not have any advertised rate limits. As the API moves towards a production release, we will update this section with specific rate limits. For now, the API will return HTTP 429 on an as-needed basis.
Polling async tasks¶
When you submit async inference tasks, poll each task's status with a delay that grows between attempts instead of a fixed tight interval. With a fixed interval, total poll traffic scales with the number of outstanding tasks: a large batch that outpaces the endpoint's drain rate generates an ever-growing poll load on the API while the tasks it is asking about cannot complete any faster.
The tenacity library (used in the backoff examples below) expresses this directly;
wait_random_exponential adds the jitter that keeps polls from many tasks from
synchronizing:
from tenacity import retry, retry_if_result, stop_after_delay, wait_random_exponential
@retry(
retry=retry_if_result(lambda response: response.status not in ("SUCCESS", "FAILURE")),
wait=wait_random_exponential(min=1, max=60),
stop=stop_after_delay(3600),
)
def wait_for_task(client, task_id):
return client.get_async_task(task_id)
For large batches, also bound how many tasks you have outstanding (submit in windows sized to the endpoint's throughput) rather than submitting everything up front and polling the whole set.
Error mitigation¶
Retrying with exponential backoff¶
One easy way to avoid rate limit errors is to automatically retry requests with a random exponential backoff. Retrying with exponential backoff means performing a short sleep when a rate limit error is hit, then retrying the unsuccessful request. If the request is still unsuccessful, the sleep length is increased and the process is repeated. This continues until the request is successful or until a maximum number of retries is reached. This approach has many benefits:
- Automatic retries means you can recover from rate limit errors without crashes or missing data
- Exponential backoff means that your first retries can be tried quickly, while still benefiting from longer delays if your first few retries fail
- Adding random jitter to the delay helps retries from all hitting at the same time.
Below are a few example solutions for Python that use exponential backoff.
Example #1: Using the tenacity library¶
Tenacity is an Apache 2.0 licensed general-purpose retrying library, written in Python, to simplify the task of adding retry behavior to just about anything. To add exponential backoff to your requests, you can use the tenacity.retry decorator. The below example uses the tenacity.wait_random_exponential function to add random exponential backoff to a request.
import llmengine
from tenacity import (
retry,
stop_after_attempt,
wait_random_exponential,
) # for exponential backoff
@retry(wait=wait_random_exponential(min=1, max=60), stop=stop_after_attempt(6))
def completion_with_backoff(**kwargs):
return llmengine.Completion.create(**kwargs)
completion_with_backoff(model="llama-2-7b", prompt="Why is the sky blue?")
Example #2: Using the backoff library¶
Backoff is another python library that provides function decorators which can be used to wrap a function such that it will be retried until some condition is met.