Policy-Driven Model Routing: Selecting the Right LLM Per Request

The first LLM integration is usually simple: you pick one provider, one model, one API key, and ship. The second and third are where things start to hurt.

Suddenly you have a "cheap" model for bulk tasks, a "smart" model for critical flows, maybe a vision model, maybe a provider change for one region. Routing logic leaks into handlers, feature flags, and tests. A simple "call the model" turns into nested conditional branches scattered across the codebase.

This post is about treating model choice as a first-class concern. We'll walk through how to build a policy-driven routing layer, an AI gateway that picks the right LLM per request based on feature, tenant tier, risk profile, latency budget, and cost constraints. Along the way we'll see how to keep it config-driven rather than hardcoded, safe with fallbacks and experiments, and observable so every routing decision is explainable. We'll ground it in a real product context, as used in Hoomanely's AI stack powering pet wellness experiences.

The problem: one "default model" doesn't scale

At small scale, a single default model is fine. But as your AI surface area grows, three constraints collide. Cost, draft emails versus safety-critical health advice shouldn't be billed at the same rate per token, and some tenants can pay for premium quality while others can't. Latency, a 2-second multi-turn assistant is acceptable, a 2-second autocomplete in a mobile app is not. Risk profile, a misphrased marketing tagline is annoying, a misworded escalation or health recommendation can be damaging.

If you route everything through a single "best" model, you either overpay by using an over-powered model for trivial tasks, or underperform by using a cheap model where you really needed reliability, context handling, or better safety tooling. This starts as a quick patch and ends as if-else spaghetti across services, with no single place to answer why a request hit that model, what the cost distribution is per feature and tenant, or whether you can turn on a canary for one segment. You need a routing layer that centralizes these decisions.

Concept: what is policy-driven model routing

Policy-driven model routing means: given a request with known attributes (feature, user, risk, budget, etc.), choose which model to call using declarative policies instead of ad-hoc code. You define policies like "for free-tier users on bulk summarization, use cheap-4k up to 512 tokens," "for enterprise incident reports, always use premium-32k and enable safety filters," or "during this canary experiment, 10% of chat_support traffic goes to candidate-8b."

The routing layer becomes an AI gateway that receives a typed, normalized inference request, evaluates policies over request attributes, picks a model or chain of models, and executes with fallbacks while logging why that decision was made. This gives you separation of concerns, app code says what task it wants and policies decide which model; easier evolution, changing routing rules is a config change, not a deploy; and better observability, you can answer "what policy fired" in one place.

Architecture: a Python-based AI gateway with routing

Imagine your stack: mobile/web apps, backend APIs (Python, Node, Go), an AI gateway service (Python-based), and multiple LLM providers (Bedrock, OpenAI, Anthropic, internal models). The AI gateway owns request normalization, policy evaluation, model client abstraction, fallbacks and experiments, and telemetry plus audit logging.

An app sends a logical request to the AI gateway:

{
  "feature": "session_summary",
  "tenant_id": "acme_inc",
  "user_tier": "pro",
  "risk_level": "medium",
  "input_size": 2300,
  "timeout_ms": 2500,
  "payload": { "...": "..." }
}

The gateway normalizes and enriches with metadata (tenant config, user tier, region, platform, historical signals like spend YTD). The policy engine evaluates rules to pick a model config (model_id, max_tokens, safety_preset). The model client executes, calling the provider and handling retries, timeouts, and fallbacks. The router logs the decision, policy ID, matched conditions, chosen model, latency, tokens, and success/failure.

Policy design: encoding product decisions as config

The heart of the system is the policy model, how you declare rules and resolve conflicts. A practical approach is tiered policy evaluation: feature-level policy defines the default behavior for a feature, tenant/tier overrides define what changes for paid plans or specific customers, risk and SLA constraints define what happens if latency or safety must be prioritized, and experiment flags handle A/B tests, canaries, or traffic splits.

A simple YAML structure might look like:

policies:
  - id: "feature.chat.default"
    when:
      feature: "chat_assistant"
    use_model: "vendor_a::chat-8k"
    max_tokens: 1024
 
  - id: "feature.chat.enterprise_upgrade"
    when:
      feature: "chat_assistant"
      tenant_tier: "enterprise"
    use_model: "vendor_b::chat-premium-32k"
    max_tokens: 2048
 
  - id: "feature.summary.low_risk_bulk"
    when:
      feature: "bulk_summary"
      risk_level: "low"
    use_model: "vendor_c::cheap-4k"
    max_tokens: 512

You can extend this with conditional expressions, like preferred models under a latency threshold. The engine should match all applicable policies, resolve with priority or specificity rules where the most specific "when" wins, and attach metadata like max tokens, safety preset, and temperature.

In multi-tenant products, routing is a powerful way to encode SLAs: free tier gets a low-cost model with stricter rate limits and lower max tokens, pro tier gets a balanced model with a moderate context window, and enterprise gets premium models, longer context, higher timeouts, and better fallbacks. Those decisions stay outside your app handlers, the handler just says what it wants.

At Hoomanely, we build AI features around pet health, nutrition, and daily routines, from lightweight explanations to deeper multi-signal insights to user-facing guidance. Policy-driven routing helps us use cost-efficient models for bulk summarization of sensor events, reserve stronger models for flows where wording and nuance matter more, and gradually introduce new model families via canaries for specific segments before rolling out broadly. We rarely want to hardcode "use Model X for EverBowl and Model Y for EverSense." Instead we route by feature category and risk level, so new devices and features can plug in without rewriting routing logic.

Implementing the policy engine

You don't need a full rule engine to get started. Create a typed request object every caller uses:

from dataclasses import dataclass
from typing import Dict, Any
 
@dataclass
class InferenceRequest:
    feature: str
    tenant_id: str
    user_tier: str
    risk_level: str
    input_size: int
    timeout_ms: int
    payload: Dict[str, Any]
    metadata: Dict[str, Any] = None

Load your YAML into normalized in-memory rules:

@dataclass
class Policy:
    id: str
    conditions: Dict[str, Any]   # e.g. {"feature": "chat_assistant", "user_tier": "enterprise"}
    config: Dict[str, Any]       # e.g. {"use_model": "vendor_a::chat-8k"}
    priority: int = 0

Simple matching can check that all key-value pairs are equal, plus helpers for ranges:

def matches(policy: Policy, req: InferenceRequest) -> bool:
    for key, expected in policy.conditions.items():
        actual = getattr(req, key, None)
        if isinstance(expected, dict) and "lte" in expected:
            if not (actual <= expected["lte"]):
                return False
        elif actual != expected:
            return False
    return True

Then evaluation:

def select_policy(policies, req: InferenceRequest) -> Policy:
    candidates = [p for p in policies if matches(p, req)]
    if not candidates:
        raise RuntimeError("No matching policy")
    # pick highest priority, then most specific (more conditions)
    candidates.sort(key=lambda p: (p.priority, len(p.conditions)), reverse=True)
    return candidates[0]

The output of the policy engine is a model invocation plan, not just a string, keeping everything explicit and testable.

Fallbacks, canaries, and A/B tests

Routing isn't just picking one model, it's also about what happens when things go wrong and how to introduce change safely. Common fallbacks include same provider with a smaller context (retry with a smaller model if the big one times out), a different provider with a similar spec, or a heuristic fallback that degrades to simpler behavior. Policies can include fallback chains, and the router's execution path tries the first model, then falls back on timeout or a specific error class, then tries an alternate provider, logging which step succeeded.

For canaries and A/B testing, introduce a traffic split policy where variants are assigned weights. The router deterministically assigns a user or session to a variant, logs variant name and chosen model, and lets you compare cost, latency, and user outcomes. Because this is config, not code, product and infra teams can collaborate without stepping on each other.

Observability: making routing decisions explainable

Routing is only useful if you can inspect it when something breaks. Every request should emit a structured decision log capturing trace_id, feature, tenant, policy_id, chosen_model, fallback_used, experiment, latency, token counts, and cost. From this you can build dashboards answering which policies fire most often, how traffic is distributed across models, where fallbacks trigger, and which experiments are succeeding. In more advanced setups you can capture input shape metrics to see where a model struggles, attach user outcomes for offline analysis, and run offline replays asking how this week's traffic would look routed to a different model.

Avoiding anti-patterns

Policy explosion happens if every small nuance becomes its own policy, drowning you in config, start with feature-level policies and introduce tenant or tier overrides only when truly needed. Hidden routing in app code means once you introduce the router, don't keep sprinkling extra conditionals in handlers, use overrides sparingly and in a structured way like a debug_force_model for QA-only workflows. No tests for policies is risky, treat policies like code with unit tests for matching, snapshot tests for golden scenarios, and startup validations checking for dangling model IDs or overlapping rules without priority.

A simple testing pattern:

def test_enterprise_chat_uses_premium():
    req = InferenceRequest(
        feature="chat_assistant",
        tenant_id="bigco",
        user_tier="enterprise",
        risk_level="medium",
        input_size=800,
        timeout_ms=3000,
        payload={}
    )
    policy = select_policy(ALL_POLICIES, req)
    assert policy.config["use_model"] == "vendor_b::chat-premium-32k"

This keeps your routing layer from silently drifting.

A practical rollout: wrap existing calls behind a router interface with no behavior change yet, introduce a simple policy file for 1-2 features reproducing existing behavior, add one new routing dimension like user tier to a small subset of tenants, add fallbacks and configure them for critical features plus one canary experiment, and gradually migrate remaining features, eventually requiring all new features to define a feature code and routing policy before shipping. Within a few iterations, "which model should we use" becomes a conversation about policies and SLAs, not copy-pasted API keys.

Key takeaways

Policy-driven model routing is less about fancy rule engines and more about good boundaries. Treat model choice as a platform concern, not a local optimization. Express routing logic as config and policies, not scattered conditionals. Support fallbacks, canaries, and A/B tests as first-class features. Invest early in observability, so every decision is explainable. And tie routing to product decisions, features, tiers, risk profiles, so your AI stack naturally reflects your business and UX priorities. As AI touches multiple surfaces, from device-driven insights to in-app coaching, this approach keeps the system adaptable, letting new models and providers get introduced safely.