LLM Routing Layer, LiteLLM, OpenRouter, Portkey
Provider lock-in is expensive. Different tool-calling workloads suit different models. Routing gateways give one API surface, retries, failover, cost tracking, and guardrails. Three archetypes dominate 2026: LiteLLM (open-source self-hosted), OpenRouter (managed SaaS), Portkey (production-grade, open-sourced in March 2026). This lesson names the decision criteria and walks a stdlib routing gateway.
Learning Objectives
- Distinguish self-hosted, managed, and production-grade routing options.
- Implement a fallback chain that retries on provider failures in a defined priority order.
- Track per-request cost and token usage across providers.
- Decide between LiteLLM, OpenRouter, and Portkey for a given production constraint.
The Problem
Scenarios where provider routing matters:
- Cost. Claude Sonnet costs 3x what Haiku costs. For a triage task, Haiku is enough; for a synthesis task, Sonnet is worth it. Route per-request.
- Failover. OpenAI has a bad hour. Every request fails. You want automatic fallback to Anthropic without redeploying.
- Latency. A live chat UI needs fast time-to-first-token. A batch summarizer does not. Route by latency SLA.
- Compliance. EU users must stay in EU regions. Route by region.
- Experimentation. A/B two models on the same workload. Route by test bucket.
Hand-coding all of this per integration is repetitive. A routing gateway gives one OpenAI-compatible API and handles the rest.
The Concept
OpenAI-compatible proxy shape
Everyone speaks OpenAI-shape. The routing gateway exposes /v1/chat/completionsaccepts the OpenAI schema, and internally proxies to Anthropic / Gemini / Cohere / Ollama / anything. The client does not care.
Model aliases
Instead of a pinned snapshot id, your code says our_smart_model. The gateway maps aliases to real models. When a provider ships a new generation, you change the alias server-side; your code does not touch a thing.
Fallback chains
primary: openai/gpt-4o
on 5xx: anthropic/claude-3-5-sonnet
on 5xx: google/gemini-1.5-pro
on 5xx: refuseGateways define this in a config. Retries count against a budget so fallback cascades do not explode cost.
Semantic caching
Identical-or-near-identical prompts hit a cache instead of the provider. Savings on repeated agent loops can be 30 to 60 percent. Keys are embedding-based; near-identical prompts share a cache slot.
Guardrails
Gateway-level:
- PII redaction. Regex or ML-based pass before sending prompts.
- Policy violations. Reject prompts with prohibited content.
- Output filters. Scrub completions for leaks.
Portkey and Kong both ship opinionated guardrails. LiteLLM leaves them optional.
Per-key rate limits
One API key = one team. Per-key budgets prevent one team from consuming the shared quota. Most gateways support this.
Self-hosted vs managed trade-offs
| Factor | LiteLLM (self-hosted) | OpenRouter (managed) | Portkey (production) |
|---|---|---|---|
| Code | Open source, Python | Managed SaaS | Open source (Mar 2026) + managed |
| Setup | Deploy a proxy | Sign up | Either |
| Providers | 100+ | 300+ | 100+ |
| Billing | Your own keys | OpenRouter credits | Your own keys |
| Observability | OpenTelemetry | Dashboard | Full OTel + PII redaction |
| Best for | Teams that want full control | Rapid prototyping | Production with compliance |
LiteLLM wins when you have an SRE team and want data sovereignty. OpenRouter wins when you want a single subscription and no infra. Portkey wins when you need guardrails and compliance out of the box.
Cost tracking
Every request carries providermodelinput_tokensoutput_tokens. Multiply by per-model per-token prices (pulled from a pricing sheet the gateway maintains). Per-user / per-team / per-project aggregation.
MCP plus routing
A gateway can route both LLM calls AND MCP sampling requests. When a sampling request's modelPreferences prefer a specific model, the gateway translates to the right backend. This is where Phase 13 · 17 (MCP gateway) and this lesson's routing gateway sometimes merge into one service.
Routing strategies
- Static priority. First in list; fall back on error.
- Load balancing. Round-robin or weighted.
- Cost-aware. Pick the cheapest model meeting latency / quality.
- Latency-aware. Pick the fastest model in the last N minutes.
- Task-aware. Prompt classifier routes coding to one model, summarization to another.
Use It
code/main.py implements a routing gateway in ~150 lines: accepts OpenAI-shaped requests, translates to per-provider stubs, runs a priority fallback chain, tracks per-request cost, and applies a PII redaction pass on inputs. Run it with three scenarios: normal request, primary-provider outage triggering fallback, PII leakage caught by redaction.
What to look at:
ROUTESdict: alias -> priority-ordered list of concrete providers.- Fallback loop retries on 5xx.
- Cost tracker multiplies token usage by per-model rates.
- PII redactor scrubs SSN-shaped patterns before forwarding.
Ship It
This lesson produces outputs/skill-routing-config-designer.md. Given a workload profile (latency, cost, compliance), the skill picks LiteLLM / OpenRouter / Portkey and produces a routing config.
Exercises
- Run
code/main.py. Trigger the outage scenario; confirm fallback lands on the second provider and cost is attributed correctly. - Add semantic caching: SHA256 of the prompt is a lookup key; cache hits return instantly. Measure cost savings on a repeated call.
- Add a prompt classifier that routes "code ..." prompts to an alias favoring intelligence and "summarize ..." prompts to an alias favoring speed.
- Design per-team budgets: each team has a monthly spend cap; gateway refuses requests once cap is hit. Pick an enforcement granularity (per-request or windowed).
- Read LiteLLM, OpenRouter, and Portkey docs side by side. Name the one feature each ships that the other two do not.
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| Routing gateway | "LLM proxy" | One-API-surface layer in front of many providers |
| OpenAI-compatible | "Speaks the OpenAI schema" | Accepts /v1/chat/completions shape, translates to any backend |
| Model alias | "our_smart_model" | Name in your code that the gateway maps to a concrete model |
| Fallback chain | "Retry list" | Ordered list of providers attempted on failure |
| Semantic caching | "Prompt-embedding cache" | Key is embedding of the prompt; near-duplicates share a cache hit |
| Guardrails | "Input/output filters" | Redact PII, reject policy violations |
| Per-key rate limit | "Team budget" | Quota scoped to an API key |
| Cost tracking | "Per-request spend" | Aggregate token usage x price per model |
| LiteLLM | "The open proxy" | Self-hostable OSS routing gateway |
| OpenRouter | "The managed SaaS" | Hosted gateway with credit-based billing |
| Portkey | "The production option" | Open-source + managed with guardrails built in |
Further Reading
- LiteLLM, docsself-hosted routing gateway
- OpenRouter, quickstartmanaged routing SaaS
- Portkey, docsproduction routing with guardrails
- TrueFoundry, LiteLLM vs OpenRouterdecision guide
- Relayplane, LLM gateway comparison 2026vendor survey
The full code
Every file for this lesson. Copy it, run it, then change it.
"""Phase 13 Lesson 21 - LLM routing gateway, stdlib.
OpenAI-compatible request in; priority fallback chain picks a backend; cost
tracker accumulates spend per-request. PII redaction runs pre-dispatch.
Backend providers are stubs. Switching one to "outage" shows fallback.
Run: python code/main.py
"""
from __future__ import annotations
import json
import re
import time
from dataclasses import dataclass, field
from typing import Callable
# cost per 1M tokens (input, output); fake rates for demo
PRICES = {
"openai/gpt-4o": (5.0, 15.0),
"openai/gpt-4o-mini": (0.15, 0.60),
"anthropic/claude-sonnet": (3.0, 15.0),
"anthropic/claude-haiku": (0.80, 4.0),
"google/gemini-pro": (1.25, 5.0),
}
OUTAGE: set[str] = set()
def provider_call(model: str, messages: list[dict]) -> dict:
if model in OUTAGE:
raise RuntimeError(f"simulated 5xx from {model}")
time.sleep(0.01)
last = messages[-1]["content"]
out_toks = max(20, len(last) // 3)
return {
"id": f"resp_{model.replace('/', '_')}",
"model": model,
"choices": [{"message": {"role": "assistant",
"content": f"[{model}] echoed: {last[:60]}"}}],
"usage": {"prompt_tokens": len(last) // 4, "completion_tokens": out_toks},
}
# aliases -> fallback chain
ROUTES = {
"smart": ["openai/gpt-4o", "anthropic/claude-sonnet", "google/gemini-pro"],
"fast": ["openai/gpt-4o-mini", "anthropic/claude-haiku"],
}
PII_PATTERNS = [
re.compile(r"\b\d{3}-\d{2}-\d{4}\b"), # SSN
re.compile(r"\b\d{16}\b"), # credit card
]
def redact_pii(text: str) -> tuple[str, bool]:
redacted = False
for pat in PII_PATTERNS:
if pat.search(text):
text = pat.sub("[REDACTED]", text)
redacted = True
return text, redacted
@dataclass
class Invocation:
alias: str
chosen_model: str = ""
attempts: list[str] = field(default_factory=list)
input_tokens: int = 0
output_tokens: int = 0
cost_usd: float = 0.0
redacted: bool = False
response: dict | None = None
error: str | None = None
def route(alias: str, messages: list[dict]) -> Invocation:
inv = Invocation(alias=alias)
# redact pii on inputs
new_msgs = []
for m in messages:
txt, r = redact_pii(m["content"])
if r:
inv.redacted = True
new_msgs.append({"role": m["role"], "content": txt})
chain = ROUTES.get(alias, [alias])
for model in chain:
inv.attempts.append(model)
try:
resp = provider_call(model, new_msgs)
inv.chosen_model = model
inv.response = resp
u = resp["usage"]
inv.input_tokens = u["prompt_tokens"]
inv.output_tokens = u["completion_tokens"]
in_rate, out_rate = PRICES.get(model, (0, 0))
inv.cost_usd = (u["prompt_tokens"] * in_rate +
u["completion_tokens"] * out_rate) / 1_000_000
return inv
except RuntimeError as e:
continue
inv.error = "all providers failed"
return inv
def demo() -> None:
print("=" * 72)
print("PHASE 13 LESSON 20 - LLM ROUTING GATEWAY")
print("=" * 72)
print("\n--- scenario 1: smart route, primary available ---")
inv = route("smart", [{"role": "user", "content": "explain MCP"}])
print(f" chosen : {inv.chosen_model}")
print(f" attempts: {inv.attempts}")
print(f" tokens : in={inv.input_tokens} out={inv.output_tokens}")
print(f" cost : ${inv.cost_usd:.6f}")
print(f" reply : {inv.response['choices'][0]['message']['content']}")
print("\n--- scenario 2: openai/gpt-4o OUTAGE -> falls back to Claude ---")
OUTAGE.add("openai/gpt-4o")
inv = route("smart", [{"role": "user", "content": "same request"}])
print(f" chosen : {inv.chosen_model}")
print(f" attempts: {inv.attempts}")
print(f" cost : ${inv.cost_usd:.6f}")
OUTAGE.clear()
print("\n--- scenario 3: PII in input gets redacted pre-dispatch ---")
inv = route("fast", [{"role": "user",
"content": "contact me at SSN 123-45-6789 please"}])
print(f" redacted: {inv.redacted}")
print(f" reply : {inv.response['choices'][0]['message']['content']}")
print("\n--- scenario 4: all providers down ---")
OUTAGE.update(ROUTES["fast"])
inv = route("fast", [{"role": "user", "content": "help"}])
print(f" attempts: {inv.attempts}")
print(f" error : {inv.error}")
if __name__ == "__main__":
demo()