Rate limits & capacity

Two different conditions return 429, and they mean different things.

CONCURRENCY_LIMIT

Civix runs on dedicated hardware with a bounded number of inference slots. When every slot is busy and the queue is full, further requests are rejected immediately rather than left to hang:

{
  "error": {
    "code": "CONCURRENCY_LIMIT",
    "type": "concurrency_limit",
    "message": "Too many concurrent inference requests. Retry shortly.",
    "request_id": "req_0b903419521e4e2c",
    "retryable": true
  }
}

This is not a quota violation. You have not exceeded an allowance; the service is momentarily saturated. A short retry usually succeeds.

Rejecting quickly is deliberate. The alternative — holding your request open until something frees up — wastes your timeout budget and tells you nothing. A fast, explicit 429 lets you back off or route elsewhere.

RATE_LIMITED

A per-key rate limit was exceeded. Honour the Retry-After header.

Handling both

Retry with exponential backoff and jitter. Jitter matters: without it, several clients rejected at the same moment retry in lockstep and re-saturate the service.

import random, time
from openai import APIStatusError

for attempt in range(4):
    try:
        response = client.chat.completions.create(...)
        break
    except APIStatusError as e:
        if e.status_code != 429 or attempt == 3:
            raise
        retry_after = e.response.headers.get("retry-after")
        delay = float(retry_after) if retry_after else (2 ** attempt) * 0.5
        time.sleep(delay + random.uniform(0, 0.3))

Reducing your chance of being limited

• Avoid many very large prompts at once. Concurrent large prompts contend for the same hardware and slow each other down considerably.
• Stream. Streamed requests are not bound by the non-streaming deadline, so they are far less likely to fail under load.
• Serialise your own batch work rather than firing everything in parallel.

Higher limits are available — get in touch and tell us about your workload.