Backpressure for LLMs: Managing Load Spikes and Token Floods in Real-Time AI Services

LLM features usually start life in a happy place: a handful of users, low traffic, and plenty of quota. Latency looks fine, tokens are cheap, logs are quiet. Then a product launch, a marketing campaign, or a new "Ask AI" button lands, and suddenly your once-stable service is drowning in concurrent chats, long prompts, and reconnect storms.

The failures are rarely dramatic. Instead you see queues growing quietly, p99 latency creeping up, memory pressure on your workers, and more "rate limit exceeded" responses from your LLM provider. Users experience it as "AI feels sluggish today" or "sometimes my chat just spins."

Backpressure is how you keep that from happening. This post walks through how to design backpressure-aware LLM backends: bounding queues, shaping bursts, enforcing token budgets, and degrading gracefully when demand exceeds capacity, all in a way that fits naturally into a modern Python-based AI backend.

The real problem: where LLM systems crack under load

Most LLM backends fail in predictable ways when traffic spikes. The tricky part is that LLM workloads aren't just "requests per second," they're tokens per second, context size per request, and session concurrency and duration.

Three common failure modes. Token floods instead of simple QPS spikes: two requests per second sounds trivial until each request is a 12k-token prompt with a 2k-token response. Your provider charges and rate-limits on tokens, not calls, so you're blowing through a tokens-per-minute cap even though QPS looks small. Slow prompts and long-lived sessions: chat UIs encourage long context histories and streaming responses, meaning CPU-bound prompt construction, network-bound streaming connections staying open for tens of seconds, and workers tied up while users slowly consume responses. Too many concurrent sessions or reconnect storms: mobile apps refresh, WebSockets reconnect, and browser tabs duplicate, when a network blip happens you get a reconnect storm, hundreds or thousands of sessions all trying to resume at once. Without backpressure these show up as saturated worker pools, unbounded queue growth, provider throttling or hard failing, and cascading timeouts between services.

A simple mental model for backpressure in LLM stacks

Backpressure is the system's ability to say "not now, or not like this," instead of silently accepting work it can't handle. A capacity contract states, at any moment, the maximum concurrent calls, tokens per second, and queued requests you can safely handle. A fairness contract states no single user, tenant, or feature is allowed to dominate capacity. A degradation contract states that when demand exceeds capacity, you'll reject fast with a clear error, or downgrade the experience with a cheaper model, shorter context, or cached answer.

For LLM systems, capacity is shaped by provider-side limits (requests/min, tokens/min, concurrent streams), your infrastructure (CPU, memory, worker pools), and product expectations (p95 latency targets, cost ceilings). Without making these explicit, you end up with accidental behavior: queues grow until something crashes.

Layered design: where to apply backpressure

Good backpressure systems are layered, not monolithic. You don't want one giant global knob, you want small, predictable controls at each layer: edge (admission control and rate limiting), service (bounded queues and worker pools), model gateway (token and concurrency caps), client (burst shaping and reconnect policies), and product (graceful degradation strategies).

At the edge, your API gateway or edge proxy is the first and cheapest place to say no. Per-tenant and per-API rate limits, like 5 requests per second and 10k tokens per minute per tenant, sliding-window or token-bucket algorithms so short bursts are allowed but sustained abuse isn't, and simple read-only checks that don't hit your DB just to reject.

You don't know the exact token count yet at the edge, but you can use historical averages per endpoint or tenant to approximate, like assuming a chat stream endpoint averages 2k prompt plus 1k completion tokens. If a tenant already burns most of its tokens-per-minute allocation, you can start rejecting new high-cost requests or allowing only small or cached operations. A tiny admission check:

def should_admit(tenant_state, endpoint):
    if tenant_state.requests_in_last_sec > TENANT_MAX_RPS:
        return False
 
    est_tokens = ENDPOINT_TOKEN_ESTIMATE[endpoint]
    if tenant_state.tokens_used_last_min + est_tokens > TENANT_TPM_LIMIT:
        return False
 
    return True

This doesn't have to be perfect, it just needs to be fast and conservative.

Inside your service: bounded queues and worker pools

Once a request passes the edge it enters your LLM service, where classic backpressure patterns from microservices apply, but tuned to long-running LLM calls. Bound your queues, if the queue is full, reject or degrade, don't just append. Limit in-flight LLM calls via a worker pool or semaphore. Separate short and long workloads to avoid starvation, like chat versus overnight batch.

MAX_INFLIGHT = 64
MAX_QUEUE = 128
 
semaphore = asyncio.Semaphore(MAX_INFLIGHT)
queue = asyncio.Queue(MAX_QUEUE)
 
async def enqueue_request(req):
    try:
        queue.put_nowait(req)
    except asyncio.QueueFull:
        raise TooBusyError("LLM backend overloaded")
 
async def worker():
    while True:
        req = await queue.get()
        try:
            async with semaphore:
                await process_llm_request(req)
        finally:
            queue.task_done()

The important part is the policy when the queue is full, reject fast with a specific error, or enqueue a degraded variant with shorter context or a cheaper model, or route to a separate fallback pipeline. If you don't make that choice explicit, your system will do the worst possible thing, silently accept work it can't finish on time.

At the model gateway: token and concurrency caps

Your LLM gateway, the component that actually calls the provider, should enforce hard limits. Max tokens per request caps prompt plus completion tokens, trimming context or truncating logs if the prompt is too long. Max concurrent calls per model or tenant recognizes that each model has different throughput and cost, allowing more concurrency for a cheaper model and less for a premium one. Tenant-level token budgets, daily, hourly, and per-session ceilings, mean when a tenant hits a budget you switch to a smaller model, enforce shorter answers, or return a quota-reached response with clear messaging.

This is also where you handle provider-side backpressure, respecting rate-limit headers, implementing retries with backoff only up to a point, and tripping a circuit breaker when the provider is unhealthy to fail fast.

On the client: burst shaping and reconnect behavior

Backpressure isn't just a server concern, your client can either amplify or smooth spikes. Deduplicate rapid inputs, don't send a new completion request on every keystroke, throttle or debounce. Keep a single active session per view, canceling old streams when starting a new one. Retry with jitter, not immediate tight loops on errors. And show errors early rather than spinning forever.

Reconnect storms are particularly nasty, imagine 10,000 devices with open streams, a network blip happens, and all reconnect within a second. Without backoff and jitter your backend sees a cliff of traffic. Teach your client to retry after a randomized small delay, respect retry-after headers from the server, and avoid re-sending the exact same query if the previous one was successfully processed.

Product-level: graceful degradation strategies

Backpressure isn't just about saying no, it's about choosing how to degrade. Cheaper or smaller models, dropping to a smaller model under heavy load or budget pressure. Shorter context or summaries, summarizing the last N messages instead of sending the full history. Reduced frequency, offering suggestions on explicit triggers instead of real-time on every keystroke. And fallback features, showing cached insights or a minimal mode if AI is unavailable.

Implementation patterns in a Python-based LLM backend

A typical stack includes an API Gateway (NGINX, API Gateway, or Envoy), a Python backend (FastAPI, Flask, or Django), an LLM gateway calling cloud LLM APIs, Redis/Dynamo/Postgres for state, and async I/O for streaming.

At the process level, maintain explicit counters:

INFLIGHT = 0
INFLIGHT_MAX = 64
QUEUE_MAX = 128
 
async def handle_chat(request):
    global INFLIGHT
    if INFLIGHT >= INFLIGHT_MAX:
        if queue_length() >= QUEUE_MAX:
            return too_busy()  # 429 with clear payload
        else:
            return enqueue_for_later(request)
 
    INFLIGHT += 1
    try:
        return await call_llm(request)
    finally:
        INFLIGHT -= 1

Track INFLIGHT per tenant and per model, expose these metrics via Prometheus or CloudWatch, and auto-tune INFLIGHT_MAX using observed latency and CPU usage.

A small token budget manager component receives estimated tokens for each request, checks against tenant or session budgets, and updates usage atomically:

class TokenBudget:
    def __init__(self, store):
        self.store = store  # e.g. Redis
 
    def check_and_reserve(self, tenant_id, est_tokens):
        used = self.store.get_tokens_last_min(tenant_id)
        if used + est_tokens > TENANT_TPM_LIMIT:
            return False
        self.store.increment_tokens(tenant_id, est_tokens)
        return True

Before calling the LLM, estimate tokens based on prompt length and max_tokens, check_and_reserve, and degrade or reject if false. Later you can replace estimates with actual token usage from provider responses.

For streaming responses (Server-Sent Events or WebSockets), stream from provider to client incrementally but keep an eye on stream duration and total tokens emitted, terminate politely when limits are reached with a clear final message, reduce max_tokens dynamically based on current load, and consider responding with a short summary plus a link to expand later.

At Hoomanely, we build pet-health experiences combining sensor data with LLM-powered insights, daily summaries, explanations of trends, and proactive nudges for pet parents. Many of these flows are real-time and session-based, like an owner opening the app in the evening to explore activity and nutrition. Typical usage spikes happen at predictable times, some features are nice-to-have while others are critical, and we need to stay within strict cost budgets while keeping core experiences responsive. We put tenant-level token budgets around long-form explanations, prioritize short high-signal alerts over heavy storytelling prompts during spikes, use bounded queues and concurrency caps in our LLM gateway, and expose rich metrics on tokens per feature, per tenant, per hour to watch how backpressure decisions play out.

What "good" looks like: outcomes and signals

Look for stable p95/p99 latency even when traffic spikes, error patterns that are intentional rather than random (more 429s or busy responses under load, fewer timeouts and mysterious 5xx), predictable costs where tokens/min plateau at a configured ceiling instead of scaling unbounded, fairness across tenants so no single tenant or feature starves others during bursts, and debriefable incidents where you can say precisely when a tenant hit its token budget, what degraded, and how latency stayed within SLO. Backpressure doesn't eliminate incidents, it makes them bounded, understandable, and fixable.

Key takeaways

Think in tokens, not just requests, design capacity around tokens per second and per minute for each tenant, model, and globally. Layer your backpressure across edge, service, model gateway, client, and product levels with simple, explicit rules. Bound queues and inflight requests, never let queues grow unbounded, reject or degrade fast. Introduce token budgets and fairness so one user or feature can't monopolize capacity and cost. Plan your degradation story ahead of time. Make clients good citizens, shaping bursts, deduping inputs, and using jittered retries. And instrument everything, queue depth, in-flight requests, tokens per minute, and degradation decisions. Treat backpressure as a first-class design concern and your LLM features will feel stable, predictable, and trustworthy even when traffic spikes and token floods hit.