Traditional API limits count requests. Model systems also need token and concurrency limits because one long prompt or streaming response can consume far more capacity than a short request. The right boundary depends on who can create load: a user, a tenant, an internal job, or a whole application.
What should an LLM rate limit measure?
Measure requests for burst protection, tokens for capacity and cost, and concurrent generations for memory pressure. A single request limit is rarely enough for an LLM workload. AI gateways often enforce these policies centrally, while LLMOps data shows which route, tenant, or prompt pattern consumes the budget.
What happens after a limit is reached?
Return a clear, retryable response for work that has not started, then apply bounded backoff. Do not blindly repeat an operation that may already have taken an external action. Rate limiting protects a service only when retries obey the same budget and execution identity. A limit response must not make a consequential action safe to repeat. Durable execution needs the same care for interrupted workflows, and model routing may offer a cheaper route only if it meets the task’s tested quality bar.