The Token Budget: Decoupling Rate Limits from Request Counts

Picture this. You ship an AI-powered “summarize document” feature. Works great in staging. Looks clean in the demo. Then one enterprise customer discovers the button and decides to shove every single contract from the last three years through GPT-4, one after another, in a tight loop. Within hours your OpenAI bill is through the roof, the API key gets rate-limited, and the other 400 tenants on your platform can’t access AI features at all.

Nobody thought to build a token-aware rate limiter.

Why Traditional Rate Limiting Doesn’t Work Here

Here’s the thing that trips people up. If you’ve built API rate limiting before (and you probably have), your instinct is to reach for something like X requests per minute per user. Simple. Battle-tested. Completely useless for LLM traffic.

Not all requests cost the same. A 50-token query and a 12,000-token document summary hit the same endpoint but consume wildly different amounts of your budget.

A user who sends 10 short questions uses maybe 2,000 tokens total. Another user who sends 3 massive document-analysis prompts burns through 40,000. Under a naive request-count limiter, both users look identical. The system sees “10 requests” vs “3 requests” and concludes the first user is more active. Meanwhile, the second user is quietly eating 20x the resources.

flowchart LR
    subgraph Same Endpoint
        direction TB
        R1["'What is our refund policy?'<br/>~80 tokens"]
        R2["'Summarize this 40-page contract<br/>and extract all liability clauses'<br/>~11,500 tokens"]
    end
    
    R1 -->|Traditional limiter sees| C1["1 request ✓"]
    R2 -->|Traditional limiter sees| C1_2["1 request ✓"]
    
    R1 -->|Token-aware limiter sees| T1["80 tokens ✓"]
    R2 -->|Token-aware limiter sees| T2["11,500 tokens ⚠️"]

    style T2 fill:#ef4444,color:#fff
    style C1 fill:#10b981,color:#fff
    style C1_2 fill:#10b981,color:#fff
    style T1 fill:#10b981,color:#fff

This is why you need to think in tokens, not requests.

The Architecture: Token Budgets in Redis

The core idea is stolen directly from how Stripe handles API rate limiting for payment processing, just adapted for the economics of LLM tokens. Every tenant gets a token budget. Every request estimates its token cost before it hits the LLM. Redis tracks consumption with sliding windows.

flowchart TD
    U([User Request]) --> GW[API Gateway]
    GW --> EST[Token Estimator]
    EST --> |"Estimated: ~3,200 tokens"| RC{Redis Check}
    
    RC -->|Budget Available| LLM[LLM API Call]
    RC -->|Budget Exhausted| DEG[Degraded Response]
    
    LLM --> ACT[Record Actual Tokens Used]
    ACT --> RED[(Redis Sliding Window)]
    
    DEG --> CACHE[(Semantic Cache)]
    DEG --> SMALL[Smaller Model Fallback]
    DEG --> REJECT[429 Too Many Requests]
    
    RED --> |"Update tenant counter"| DONE([Response to User])
    CACHE --> DONE
    SMALL --> DONE
    REJECT --> DONE

    style RC fill:#f59e0b,color:#000
    style DEG fill:#ef4444,color:#fff
    style LLM fill:#8b5cf6,color:#fff
    style RED fill:#3b82f6,color:#fff

Let’s build this piece by piece.

Step 1: The Token Estimator

You cannot wait until after the LLM responds to find out how many tokens you just spent. By then the money is gone. You need to estimate before the request ever leaves your infrastructure.

OpenAI uses tiktoken for tokenization. It’s the exact same tokenizer their API uses internally, so the estimates are near-exact for the input side. Output tokens are trickier (you don’t know the response length yet), but a conservative multiplier works well enough in practice.

# middleware/token_estimator.py

import tiktoken

# Cache the encoder. Loading it on every request is surprisingly expensive.
# NOTE: This tokenizer is specific to OpenAI models. If you route to
# Anthropic or other providers, their tokenizers diverge 15-25% on
# typical prompts. You'd need a provider-aware estimator in that case.
_encoder = tiktoken.encoding_for_model("gpt-4o")

def estimate_request_tokens(
    messages: list[dict],
    max_output_tokens: int = 1024,
) -> int:
    """
    Estimate total token cost of a chat completion request.
    
    We count exact input tokens via tiktoken, then add the
    max_output_tokens as a worst-case reservation.
    """
    input_tokens = 0
    for msg in messages:
        # Every message has overhead: role tokens + content + separators
        input_tokens += 4  # <im_start>{role}\n ... <im_end>\n
        input_tokens += len(_encoder.encode(msg["content"]))
    
    input_tokens += 2  # priming tokens for the assistant reply
    
    # Reserve the full output window. Yes, this over-counts.
    # In practice, max_output_tokens over-estimates by 40-70% for typical
    # conversational responses. That's phantom budget you're locking up.
    # The reconciliation step refunds it, but between reservation and
    # settlement, other requests see a tighter budget than reality.
    # Better to over-reserve and refund than to blow the budget.
    return input_tokens + max_output_tokens


def count_actual_tokens(response) -> int:
    """After the LLM responds, grab the real count from the response."""
    return response.usage.total_tokens

That max_output_tokens reservation is a deliberate choice. You’re temporarily locking tokens that might not all get used, then reconciling after the response comes back. It’s the same pattern credit card companies use: authorize the full amount, settle the actual charge later.

Step 2: The Sliding Window in Redis

A fixed window counter (reset every 60 seconds) has a nasty edge case. A user can dump their entire budget in the last second of one window, then immediately dump another full budget in the first second of the next window. That’s 2x your intended limit in two seconds.

Sliding windows fix this. The idea is straightforward: instead of hard boundaries, you weight the previous window’s usage against the current window’s elapsed time.

# limiter/sliding_window.py

import time
import redis

r = redis.Redis(host="localhost", port=6379, db=0, decode_responses=True)

# Tier definitions (tokens per minute)
TIER_LIMITS = {
    "free":       5_000,
    "starter":   30_000,
    "business":  90_000,
    "enterprise": 300_000,
}

WINDOW_SIZE = 60  # seconds


def check_and_reserve(tenant_id: str, tier: str, estimated_tokens: int) -> dict:
    """
    Atomic check-and-reserve using a Redis Lua script.
    Returns whether the request is allowed and how much budget remains.
    """
    limit = TIER_LIMITS[tier]
    now = time.time()
    current_window = int(now // WINDOW_SIZE)
    previous_window = current_window - 1
    window_elapsed_pct = (now % WINDOW_SIZE) / WINDOW_SIZE

    # Keys for current and previous windows
    curr_key = f"tkn:{tenant_id}:{current_window}"
    prev_key = f"tkn:{tenant_id}:{previous_window}"

    # Lua script: atomic read + conditional write in one round trip
    lua_script = """
    local prev_count = tonumber(redis.call('GET', KEYS[1]) or '0')
    local curr_count = tonumber(redis.call('GET', KEYS[2]) or '0')
    local window_elapsed = tonumber(ARGV[1])
    local limit = tonumber(ARGV[2])
    local requested = tonumber(ARGV[3])
    local ttl = tonumber(ARGV[4])
    
    -- Weighted sum: previous window's contribution decays linearly
    local weighted_count = prev_count * (1 - window_elapsed) + curr_count
    
    if weighted_count + requested > limit then
        return {0, weighted_count, limit}
    end
    
    -- Reserve the tokens (tenant counter)
    redis.call('INCRBY', KEYS[2], requested)
    redis.call('EXPIRE', KEYS[2], ttl)
    
    -- Also bump the global counter atomically (O(1) reads later)
    local global_key = 'global:tkn:' .. ARGV[5]
    redis.call('INCRBY', global_key, requested)
    redis.call('EXPIRE', global_key, ttl)
    
    return {1, weighted_count + requested, limit}
    """

    result = r.eval(
        lua_script,
        2,  # number of keys
        prev_key, curr_key,
        window_elapsed_pct, limit, estimated_tokens, WINDOW_SIZE * 2, current_window,
    )

    allowed, current_usage, max_limit = result
    return {
        "allowed": bool(allowed),
        "tokens_used": int(current_usage),
        "tokens_limit": int(max_limit),
        "tokens_remaining": int(max_limit - current_usage),
        "retry_after": None if allowed else WINDOW_SIZE * (1 - window_elapsed_pct),
    }


# Reconciliation needs its own Lua script. A plain DECRBY can race
# below zero if many requests reconcile at the same time.
_lua_reconcile = """
local current = tonumber(redis.call('GET', KEYS[1]) or '0')
local refund = tonumber(ARGV[1])
local new_val = math.max(0, current - refund)
redis.call('SET', KEYS[1], new_val)
if new_val > 0 then
    redis.call('EXPIRE', KEYS[1], tonumber(ARGV[2]))
end
return new_val
"""


def reconcile_tokens(tenant_id: str, estimated: int, actual: int):
    """
    After the LLM responds, adjust the counter.
    If we over-reserved, give the difference back.
    Uses a Lua script with a floor at zero to prevent
    concurrent reconciliations from decrementing below 0.
    """
    # Note: global counter is not reconciled on refund.
    # It intentionally runs high as a conservative ceiling estimate.
    # True utilization is slightly lower than get_global_usage() reports.
    diff = estimated - actual
    if diff > 0:
        current_window = int(time.time() // WINDOW_SIZE)
        key = f"tkn:{tenant_id}:{current_window}"
        r.eval(_lua_reconcile, 1, key, diff, WINDOW_SIZE * 2)

Why a Lua script? Because this entire check-and-reserve operation needs to be atomic. If two requests from the same tenant arrive simultaneously and you do the read and write as separate Redis commands, both might read the same counter value, both conclude there’s enough budget, and both proceed. You’ve just allowed 2x the intended tokens. The Lua script runs inside Redis itself, so it’s single-threaded and race-condition-proof.

Step 3: The Middleware That Ties It Together

# middleware/rate_limit_middleware.py

from fastapi import Request, HTTPException
from fastapi.responses import JSONResponse
from limiter.sliding_window import check_and_reserve, reconcile_tokens
from middleware.token_estimator import estimate_request_tokens, count_actual_tokens


async def rate_limit_middleware(request: Request, call_next):
    tenant_id = request.state.tenant_id
    tier = request.state.tenant_tier

    # Parse the body to estimate tokens
    body = await request.json()
    messages = body.get("messages", [])
    max_output = body.get("max_tokens", 1024)

    estimated = estimate_request_tokens(messages, max_output)

    # Check budget
    budget = check_and_reserve(tenant_id, tier, estimated)

    if not budget["allowed"]:
        return JSONResponse(
            status_code=429,
            content={
                "error": "token_budget_exceeded",
                "message": f"Token budget exhausted. Resets in {budget['retry_after']:.1f}s",
                "tokens_used": budget["tokens_used"],
                "tokens_limit": budget["tokens_limit"],
            },
            headers={
                "Retry-After": str(int(budget["retry_after"])),
                "X-TokenLimit-Limit": str(budget["tokens_limit"]),
                "X-TokenLimit-Remaining": str(budget["tokens_remaining"]),
                "X-TokenLimit-Reset": str(int(budget["retry_after"])),
            },
        )

    # Let the request through
    response = await call_next(request)

    # Reconcile: refund over-reserved tokens
    if hasattr(response, "llm_usage"):
        actual = count_actual_tokens(response.llm_usage)
        reconcile_tokens(tenant_id, estimated, actual)
    else:
        # If the downstream handler doesn't attach usage stats, over-reservations
        # are never refunded, leading to silent phantom token accumulation.
        import logging
        logging.getLogger("limiter").warning(
            f"Reconciliation skipped for tenant {tenant_id}: response missing 'llm_usage'"
        )

    # Attach rate limit headers so the client can self-throttle
    response.headers["X-TokenLimit-Limit"] = str(budget["tokens_limit"])
    response.headers["X-TokenLimit-Remaining"] = str(budget["tokens_remaining"])
    return response

Those X-TokenLimit-* headers aren’t just being polite. They’re load-bearing. A well-built client SDK reads those headers and backs off gracefully before it ever gets a 429. Stripe does this. GitHub does this. If you skip the headers, you’re forcing every client to learn your limits the hard way: by crashing into them.

The Global Budget Problem

So we’ve got per-tenant limits working. Great. But there’s a second constraint that most people forget about entirely.

OpenAI gives your entire organization a global rate limit. Right now, a typical org might get 90,000 TPM (tokens per minute) across all of GPT-4o. You could have 50 tenants, each with a perfectly reasonable 5,000 TPM budget, and the math still doesn’t work. 50 × 5,000 = 250,000 TPM. You’re oversubscribed by nearly 3x.

flowchart TD
    subgraph Tenants ["50 Tenants (250K TPM allocated)"]
        T1["Tenant A: 5K TPM"]
        T2["Tenant B: 5K TPM"]
        TN["Tenant C...N: 5K TPM each"]
    end

    subgraph Global ["OpenAI Org Limit"]
        GL["90K TPM Hard Ceiling"]
    end

    subgraph Controller ["Global Budget Controller"]
        GC["Tracks real-time org-wide usage"]
        CB{Circuit Breaker}
    end

    T1 & T2 & TN --> GC
    GC --> CB
    CB -->|Under 80%| GL
    CB -->|Over 80%| THROTTLE["Throttle new requests"]
    CB -->|Over 95%| SHED["Shed low-priority traffic"]
    
    style GL fill:#ef4444,color:#fff
    style CB fill:#f59e0b,color:#000
    style SHED fill:#ef4444,color:#fff

This is the exact same problem airlines deal with. They overbook seats because they know not every passenger shows up. Most of the time, it works. But when a holiday weekend hits and everyone actually boards the plane, someone’s getting bumped.

Practical rule of thumb: the sum of your per-tenant limits should not exceed 2-3x your global ceiling. Track your p95 concurrent tenant count, not your total tenant count. If you have 500 tenants but only 40 are ever active simultaneously, you have much more room than the raw multiplication suggests.

# limiter/global_budget.py

import time
import redis

r = redis.Redis(host="localhost", port=6379, db=0, decode_responses=True)

GLOBAL_TPM_LIMIT = 90_000
THROTTLE_THRESHOLD = 0.80   # start slowing down at 80%
SHED_THRESHOLD = 0.95       # start dropping low-priority requests at 95%
WINDOW_SIZE = 60


def get_global_usage() -> int:
    """
    Read the global counter. O(1) because the tenant-level Lua script
    bumps this key atomically on every reservation.
    """
    current_window = int(time.time() // WINDOW_SIZE)
    return int(r.get(f"global:tkn:{current_window}") or 0)


def global_admission_check(
    tenant_tier: str,
    estimated_tokens: int,
    request_priority: str = "normal",
) -> dict:
    """
    Second-layer check: even if the tenant has budget,
    we might need to protect the global ceiling.
    """
    current_usage = get_global_usage()
    utilization = current_usage / GLOBAL_TPM_LIMIT

    # Under 80%: everyone gets through
    if utilization < THROTTLE_THRESHOLD:
        return {"allowed": True, "utilization": utilization}

    # 80-95%: only allow high-priority or small requests
    if utilization < SHED_THRESHOLD:
        if request_priority == "high" or estimated_tokens < 500:
            return {"allowed": True, "utilization": utilization, "throttled": True}
        # Non-critical large requests get queued, not rejected
        return {
            "allowed": False,
            "reason": "global_throttle",
            "utilization": utilization,
            "retry_after": 10,
        }

    # Over 95%: enterprise tier only, everything else is shed
    if tenant_tier == "enterprise":
        return {"allowed": True, "utilization": utilization, "throttled": True}

    return {
        "allowed": False,
        "reason": "global_capacity",
        "utilization": utilization,
        "retry_after": 30,
    }

Notice the tiered shedding. When global capacity gets tight, free-tier users sending big requests are the first to get queued. Enterprise tenants are the last to get restricted. This isn’t unfair, it’s the economics of the business. The enterprise customer paying $50K/year should absolutely get priority over a free trial account summarizing blog posts.

Graceful Degradation: Don’t Just Reject, Adapt

Here’s where most rate limiting implementations stop. Tenant over budget? 429. Done.

That’s lazy. And it’s a terrible user experience.

When someone hits their limit, you have options that are way better than a raw HTTP error. The goal is to keep the feature feeling alive even when the premium path is blocked.

flowchart TD
    REQ([Incoming Request]) --> TL{Tenant Budget?}
    
    TL -->|Available| GL{Global Budget?}
    TL -->|Exhausted| DEG[Degradation Pipeline]
    
    GL -->|Available| LLM["GPT-4o (Full Model)"]
    GL -->|Throttled| DEG
    
    DEG --> S1{Semantic Cache Hit?}
    S1 -->|Yes| CACHED["Return Cached Answer<br/>(0 tokens, ~5ms)"]
    S1 -->|No| S2{Smaller Model Available?}
    
    S2 -->|Yes| SMALL["GPT-4o-mini Fallback<br/>(~10x cheaper)"]
    S2 -->|No| S3{Can Queue?}
    
    S3 -->|Yes| QUEUE["Queue for Later<br/>(Process when budget resets)"]
    S3 -->|No| REJECT["429 with Retry-After"]

    style LLM fill:#8b5cf6,color:#fff
    style CACHED fill:#10b981,color:#fff
    style SMALL fill:#3b82f6,color:#fff
    style QUEUE fill:#f59e0b,color:#000
    style REJECT fill:#ef4444,color:#fff

# degradation/fallback_chain.py

from semantic_cache import SemanticCache
from llm_clients import gpt4o_client, gpt4o_mini_client

cache = SemanticCache(
    redis_url="redis://localhost:6379",
    similarity_threshold=0.92,
    ttl_seconds=3600,
)


async def handle_with_fallback(
    messages: list[dict],
    tenant_id: str,
    budget_status: dict,
) -> dict:
    query = messages[-1]["content"]
    
    # Tier 1: Try the semantic cache first (always, even when budget is fine)
    cached = await cache.lookup(query, namespace=tenant_id)
    if cached and cached["similarity"] > 0.92:
        return {
            "content": cached["response"],
            "model": "cache",
            "tokens_used": 0,
            "latency_ms": cached["lookup_ms"],
            "cache_hit": True,
        }

    # If budget is available, go straight to the primary model
    if budget_status["allowed"]:
        response = await gpt4o_client.chat(messages=messages)
        # Backfill the cache for future requests
        await cache.store(query, response.content, namespace=tenant_id)
        return {
            "content": response.content,
            "model": "gpt-4o",
            "tokens_used": response.usage.total_tokens,
            "cache_hit": False,
        }

    # Budget exhausted. Tier 2: try a cheaper model
    # GPT-4o-mini is roughly 10x cheaper per token
    try:
        response = await gpt4o_mini_client.chat(messages=messages)
        await cache.store(query, response.content, namespace=tenant_id)
        return {
            "content": response.content,
            "model": "gpt-4o-mini",
            "tokens_used": response.usage.total_tokens,
            "degraded": True,
            "reason": "budget_exhausted, fell back to smaller model",
        }
    except Exception:
        pass

    # Tier 3: everything is on fire, just reject cleanly
    return {
        "content": None,
        "error": "token_budget_exceeded",
        "retry_after": budget_status.get("retry_after", 60),
    }

That semantic cache at the top of the chain is doing double duty. When budget is healthy, it saves money by short-circuiting repeat questions. When budget is exhausted, it becomes your primary serving layer. In a well-tuned deployment, the cache can realistically handle 30-40% of all requests, which means that chunk of your traffic never touches the LLM at all.

There is a subtle tracking gap in this fallback path: when we redirect a rate-limited request to gpt-4o-mini, those fallback tokens are served outside the normal budget checks. Because the tenant has already exhausted their budget window, you cannot record these fallback tokens to their Redis counter without blocking them or creating complex negative budget balances. Furthermore, if you exclude them from the global counter, your billing logs and overall consumption reporting will miss them. In production, the solution is to log these degraded usages directly to a separate metric counter (or stream them directly to your billing queue) so that they are still tracked for cost attribution, even if they bypass the active rate-limiting window.

Observability: You Can’t Manage What You Can’t See

Token budgets are pointless if you can’t see them in action. You need dashboards that answer these questions in real time:

Which tenants are closest to their limits right now?
How close are we to the global OpenAI ceiling?
How often are we falling back to cheaper models vs. outright rejecting?
What’s our cache hit rate, and is it trending up or down?

# observability/metrics.py

from prometheus_client import Counter, Histogram, Gauge

# Per-tenant token consumption
tokens_consumed = Counter(
    "ai_tokens_consumed_total",
    "Total tokens consumed by tenant",
    ["tenant_id", "model", "tier"],
)

# Rate limit decisions
rate_limit_decisions = Counter(
    "ai_rate_limit_decisions_total",
    "Rate limiting outcomes",
    ["decision", "reason"],
    # decision: allowed, throttled, rejected
    # reason: tenant_budget, global_throttle, global_capacity
)

# Degradation events
degradation_events = Counter(
    "ai_degradation_events_total",
    "Fallback events when primary model unavailable",
    ["fallback_type"],
    # fallback_type: cache_hit, smaller_model, queued, rejected
)

# Token budget utilization (for Grafana gauges)
tenant_budget_utilization = Gauge(
    "ai_tenant_budget_utilization_ratio",
    "Current token budget utilization per tenant",
    ["tenant_id", "tier"],
)

global_budget_utilization = Gauge(
    "ai_global_budget_utilization_ratio",
    "Current global token budget utilization",
)

# Latency by serving path
response_latency = Histogram(
    "ai_response_latency_seconds",
    "Response latency bucketed by serving path",
    ["serving_path"],  # primary, cache, fallback_model
    buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0],
)

The metric I watch most closely is ai_degradation_events_total. If the smaller_model counter is climbing steadily week over week, that’s telling me either the tenant limits are too tight, or actual usage is growing and I need to negotiate higher limits from OpenAI (or prepurchase committed throughput). The number tells you when to act before users start complaining.

Handling the Edge Cases That Will Ruin Your Week

The Streaming Problem

If you’re using server-sent events for streaming responses (and you should be, the UX difference is massive), you have a problem. You can’t count response tokens before the stream starts. And you can’t cut off the stream mid-sentence without the user seeing garbled output.

The approach that works: reserve the full max_tokens budget up front, then reconcile after the stream finishes.

# streaming/token_accounting.py

async def stream_with_accounting(
    messages: list[dict],
    tenant_id: str,
    max_tokens: int = 1024,
):
    estimated = estimate_request_tokens(messages, max_tokens)
    
    # get_tier is an internal helper that queries your metadata store / DB 
    # to retrieve the tenant's current pricing plan or rate limit tier.
    tenant_tier = get_tier(tenant_id)
    budget = check_and_reserve(tenant_id, tenant_tier, estimated)
    
    if not budget["allowed"]:
        yield {"error": "budget_exceeded", "retry_after": budget["retry_after"]}
        return

    actual_output_tokens = 0
    
    async for chunk in gpt4o_client.stream(messages=messages, max_tokens=max_tokens):
        actual_output_tokens += count_chunk_tokens(chunk)
        yield chunk

    # Stream is done. Refund the difference.
    actual_total = estimate_request_tokens(messages, 0) + actual_output_tokens
    reconcile_tokens(tenant_id, estimated, actual_total)

The Burst Problem

A user sends 15 requests in 200 milliseconds. Each individual request is small (200 tokens), well within budget. But they all hit the estimator before any of them reach the reconciliation step, so the sliding window hasn’t caught up yet.

This is where you layer a simple request-rate limiter on top of the token budget. Not as the primary defense, just as a burst dampener.

# limiter/burst_dampener.py

def check_burst_limit(tenant_id: str, tier: str) -> bool:
    """
    Simple fixed-window request counter. 
    Not for budget management, just to prevent
    thundering herd on the token estimator.
    """
    max_concurrent = {
        "free": 3,
        "starter": 10,
        "business": 25,
        "enterprise": 50,
    }
    
    key = f"burst:{tenant_id}"
    # Use SET NX EX to atomically create the key with TTL.
    # If the key already exists, INCR it. This avoids the classic
    # race where a crash between INCR and EXPIRE leaves the key
    # immortal, permanently locking out the tenant.
    if r.set(key, 1, nx=True, ex=1):
        return True  # first request in this window
    
    current = r.incr(key)
    return current <= max_concurrent[tier]

It’s crude. That’s the point. You don’t need sophistication here, you need a circuit breaker that keeps 15 simultaneous requests from all passing the token check before any of them are recorded.

The Full Picture

flowchart TD
    subgraph Client ["Client Layer"]
        U([User / SDK])
    end

    subgraph Gateway ["API Gateway"]
        AUTH[Auth + Tenant Resolution]
        BURST[Burst Dampener]
        EST[Token Estimator]
        TL[Tenant Budget Check<br/>Redis Sliding Window]
        GL[Global Budget Check]
    end

    subgraph Serving ["Serving Layer"]
        CACHE[(Semantic Cache)]
        PRIMARY["GPT-4o"]
        FALLBACK["GPT-4o-mini"]
        QUEUE[/Request Queue/]
    end

    subgraph Data ["Data Layer"]
        RED[(Redis Token Counters)]
        PROM[Prometheus Metrics]
        GRAF[Grafana Dashboard]
    end

    U --> AUTH
    AUTH --> BURST
    BURST -->|Pass| EST
    BURST -->|Too many| R429_1["429 Burst Limit"]
    
    EST --> TL
    TL -->|Budget OK| GL
    TL -->|Exhausted| CACHE
    
    GL -->|Clear| PRIMARY
    GL -->|Throttled| CACHE
    
    CACHE -->|Hit| U
    CACHE -->|Miss| FALLBACK
    FALLBACK -->|Fail| QUEUE
    QUEUE -->|Full| R429_2["429 + Retry-After"]
    
    PRIMARY --> RED
    FALLBACK --> RED
    RED --> PROM
    PROM --> GRAF

    style PRIMARY fill:#8b5cf6,color:#fff
    style CACHE fill:#10b981,color:#fff
    style FALLBACK fill:#3b82f6,color:#fff
    style RED fill:#3b82f6,color:#fff
    style R429_1 fill:#ef4444,color:#fff
    style R429_2 fill:#ef4444,color:#fff
    style GRAF fill:#64748b,color:#fff

The Tradeoffs, Honestly

Consideration	No Rate Limiting	Request-Count Limiting	Token-Budget Limiting
Cost Protection	None	Partial (doesn’t account for token variance)	Strong
Fairness Across Tenants	Nonexistent	Roughly fair	Precisely fair
Implementation Complexity	Zero	Low	Medium-High
User Experience on Limit	Everyone suffers when budget runs out	Hard rejections	Graceful degradation
Observability	Flying blind	Basic	Full cost attribution
Operational Overhead	None	Minimal	Redis + monitoring + tier management

Nothing here is free. The token estimator adds latency to every request (though we’re talking ~1ms for tiktoken encoding, so it’s noise). The Redis Lua scripts add a network round trip. The reconciliation step means your counters are slightly inaccurate between the initial reservation and the final settlement. The degradation chain adds code paths that all need testing.

And there’s the elephant in the room: Redis is now a single point of failure for your entire AI gateway. If Redis goes down, no request can check its budget, and you’re stuck choosing between “reject everything” and “allow everything unmetered.” In production you’d want a Redis Sentinel or Cluster setup with a local in-process fallback (a simple in-memory counter with a conservative limit) so the system degrades to approximate rate limiting rather than going fully blind.

One more distinction worth keeping clean: rate limiting and cost attribution are related but they serve different masters. Rate limiting prevents overuse in real time. Cost attribution tells you which tenant to bill at the end of the month. They share the same Redis counters, but the billing pipeline reads from a durable log (Kafka, a CDC stream from Postgres) rather than ephemeral sliding window keys. Don’t conflate them.

What Comes Next

This covers the runtime rate limiting layer. The natural next steps, roughly in the order you’ll need them:

Alerting at 80% budget. Ping the tenant admin before they hit the wall, not after. This is a one-day integration with your metrics pipeline and it prevents the majority of angry support tickets.
Provider routing. If you’re running OpenAI + Anthropic + a self-hosted model, you need a routing layer that shifts traffic between providers based on cost, latency, and remaining capacity. This also solves the tokenizer divergence problem (each provider gets its own estimator).
Billing integration. Map token consumption to actual dollars on the invoice. This reads from a durable event log, not the ephemeral Redis counters.
Pre-paid credits. Tenants buy token pools in advance. The sliding window becomes a daily/monthly budget check instead of per-minute.

But the sliding window token budget described here is the foundation. Every one of those extensions builds on top of the same Redis counters and the same estimation/reconciliation loop. Get this layer right, and the rest is plumbing.

One thing I keep coming back to: the hardest part of this entire system isn’t the Lua scripts or the degradation chain. It’s deciding what the right budget numbers are for each tier. Set them too tight and your power users feel throttled on day one. Set them too loose and you’re back to the original problem. The only way through is to ship conservative limits, watch the metrics, and adjust. There’s no formula that replaces observing real usage patterns.