API Rate Limits Explained
Understand RPM, TPM, concurrency, burst limits, queues, retries, and fair-use rules for API products.
Rate limits protect both the platform and the customer
An API rate limit defines how quickly a caller can use an API. Good limits prevent abuse, protect upstream providers, keep bills predictable, and make performance fair across customers. Bad limits are confusing, too strict, or unrelated to the real bottleneck.
The most common mistake is setting a request limit while ignoring token or payload size. For AI APIs, one huge request can be more expensive than many small requests, so request-per-minute limits should be paired with token-per-minute limits.
Important rate-limit terms
Rate limits are usually described with a handful of terms. Keep these terms visible in your documentation and error responses so developers can recover correctly.
| Term | Meaning | Typical response |
|---|---|---|
| RPM | Requests per minute | 429 when too many calls arrive |
| TPM | Tokens per minute | 429 when token budget is exceeded |
| Burst | Short spike allowance | Small temporary buffer |
| Concurrency | Requests in flight at once | Queue or reject excess work |
| Retry-after | When to try again | Header with seconds or timestamp |
A simple capacity check
Assume an API has a 3,000 RPM limit and a 1,000,000 TPM limit. If 250 active users each make 0.5 requests per minute, demand is 125 RPM. If each request averages 900 tokens, demand is 112,500 TPM. Both are under the limit, so the system should be safe under normal conditions.
Queues are useful when bursts are normal
A queue helps absorb short bursts without immediately failing user requests. It is not a substitute for a business rule. If demand is permanently above your RPM or TPM budget, the queue will only hide the problem until latency becomes unacceptable.
Use a queue when requests can wait, such as background enrichment, exports, or batch analysis. For chat, search, and interactive workflows, prefer smaller queues, clear retry messages, and per-user limits that prevent one customer from consuming the shared budget.
How to design fair user limits
Start from the provider limit, subtract an operational reserve, and divide the rest across expected active users. Then apply plan tiers. Free plans should get smaller limits and stricter burst controls. Paid plans can receive higher sustained limits or dedicated capacity.
- Return structured 429 errors with limit name, current usage, and retry time.
- Track both accepted and rejected requests.
- Alert on sustained utilization, not just single-limit errors.