Best LLM Gateway 2026: Portkey, OpenRouter, LiteLLM
Five gateways took most of the production traffic this year. Routing, cost, and governance compared, plus the honest ops bill behind the cheap option.
If your agent stack still talks straight to provider SDKs, you will regret it within a quarter. Not because the raw anthropic and openai clients stopped working. They are fine. The problem is everything around the call: failing over when one provider throttles you mid-incident, attributing cost across forty agents, redacting PII before it reaches a vendor, keeping an audit trail, and swapping models without a deploy.
That is what a gateway does. It sits between your application and however many providers you have, and turns “which model are we calling” from a code change into a routing decision.
By spring the category had consolidated onto five products. This is what I would pick and why, re-checked against vendor docs on 21 August 2026.
Why the gateway showed up
A year ago you could hit one provider directly and be done. Now a normal stack has Claude Opus 5 for hard reasoning, Sonnet 5 for cheap throughput, GPT-5.5 somewhere, Gemini for the long-context jobs, and probably DeepSeek or Qwen doing batch work on an open-weights host.
That is five providers minimum. Five SDKs, five auth patterns, five error semantics, five sets of rate-limit headers, and no shared view of any of it. When one of them has a regional outage, and one of them will, your fallback runs through application code somebody wrote at 2am and nobody has tested since.
A gateway centralizes that. One endpoint, one schema (almost always OpenAI-compatible), one place for retries, fallbacks, caching, guardrails, cost tracking, and audit.
Four jobs, and no gateway does all four equally well: routing and fallback, cost and traces, governance, and caching. Which of the four you actually need is the whole decision.
OpenRouter: the easiest thing that works
One key, one OpenAI-compatible endpoint, hundreds of hosted models you swap by changing a string. Point your existing client at https://openrouter.ai/api/v1, change the model to anthropic/claude-opus-5, done. Ten minutes.
The catch is the fee. OpenRouter charges 5.5% when you buy credits. Bring your own provider key and you get a plan-dependent free allowance measured in list-price inference cost, with 5% above it. They changed the BYOK terms in August, so check the current page rather than trusting any blog post, this one included.
At $500 a month of inference that fee is invisible. At $50,000 a month it is real money for what is, at bottom, a proxy.
It is also thin on governance. Cost tracking and basic analytics exist. Redaction, prompt versioning, and serious guardrails are not the product. OpenRouter is a router, and it is a very good one.
I would use it for prototypes, for solo work, and for anything that genuinely benefits from cheap experimentation across many models. I would not put a six-figure spend behind it without doing the fee arithmetic first.
Portkey: the production one, and now Apache 2.0
Portkey was already the strongest managed gateway by late 2025: guardrails, prompt versioning, PII redaction, jailbreak detection, fallback graphs, semantic caching. In March 2026 they open-sourced the production gateway core under Apache 2.0, and pulled features that used to need a subscription (circuit breaker, semantic cache, budget limits, model catalog, config management) into the open-source build.
That matters more than it sounds. You can self-host the actual proxy in your VPC, keep traffic inside your perimeter, and pay only for the managed control plane: analytics, prompt library, evals, dashboard. For anyone under HIPAA or the EU AI Act that is the difference between a procurement deny-list and a signed contract.
The fallback configuration is genuinely expressive. Conditional fallbacks by error type, weighted load balancing, semantic caching, per-route guardrails, prompt versions with A/B routing and rollback.
The cost is complexity. The configuration surface is large enough that somebody on the team has to own it, and the docs are dense. This is a platform, not a set-and-forget proxy.
It is my default pick for a team running production agents that cares about governance.
LiteLLM: cheapest, if you can staff it
LiteLLM is the open-source Python proxy in front of a hundred-odd providers, exposed as one OpenAI-compatible API. You host it. You pay no margin to anyone, so your inference cost is the provider’s rate and nothing else.
Above roughly $10,000 a month the math gets loud. What you get for it: routing, fallbacks, virtual keys for per-team or per-customer attribution, budget enforcement, exact and semantic caching, and clean integrations with Langfuse, Datadog, and most observability stacks. What you do not get: the polished UI, the prompt-versioning experience, the curated guardrail library, or anyone to call.
The honest trade is ops capacity. Postgres for state, Redis for cache, a Helm chart, someone who notices when it degrades. If your team already runs Python services in production you will be fine, and you will be fine cheaply. If your team outsources infrastructure to a PaaS on purpose, the operational cost eats the savings. Know which team you are before you commit.
Cloudflare AI Gateway: obvious if you are already there
If your stack runs on Cloudflare this is close to a non-decision. There is a free tier, edge caching, and the tightest integration you will find with Workers, Vectorize, and R2.
It has also moved faster this year than the April version of this post gave it credit for. Guardrails have been in since early 2025, DLP screens for personal and financial data, gateway-level automatic retries with configurable backoff landed in April, and in May the whole thing moved onto a REST API on api.cloudflare.com with OpenAI-compatible, Responses-compatible, and Anthropic-compatible endpoints, plus unified billing so you are not juggling provider keys at all.
Where it still lags Portkey is prompt management, which is essentially not a feature, and the depth of the governance configuration. It is an excellent infrastructure layer. It is not a prompt platform.
For a Cloudflare shop, use it. For everyone else, it is not a reason to migrate your stack to Cloudflare.
Kong AI Gateway: the enterprise answer
This is what happens when an established API-management vendor extends into the AI lane. Semantic caching, RBAC, governance plugins, rate limiting that holds under load, and AI traffic managed in the same control plane as the rest of your services.
If your organization already runs Kong, and a surprising number do without any individual team knowing, adding the AI plugins is the path of least resistance. No new vendor, no procurement fight, and the audit story is the one your security team already approved.
If you are not already on Kong, this is not where you start.
Worth a sentence each
Helicone is observability-first, a passthrough proxy with good cost tracking and request inspection, not a routing or policy layer. Pair it, do not rely on it alone.
OpenPipe does fine-tune-aware routing, sending easy requests to a small tuned model and hard ones to a frontier model. Interesting once you are at the scale where fine-tuning pays for itself.
TrueFoundry, Eden AI, and llmgateway.io are credible smaller plays worth a look if a specific compliance requirement rules out the leaders.
Bedrock is the native option for an AWS-only shop, but it is a model marketplace with routing attached rather than a gateway, and it does not help when you also need to call something outside AWS.
What it actually costs
Sticker price is the wrong number. Total cost is provider bill plus gateway fee plus ops time.
At $1,000 a month of inference the gateway is a rounding error. Pick the developer experience you like.
At $10,000, OpenRouter’s fee is real but not decisive, Portkey’s managed plan is competitive, and LiteLLM saves most of that if you can absorb maybe half a day a week of someone’s attention.
At $100,000, the fee alone pays for an engineer. Self-hosted LiteLLM plus managed Langfuse runs well under $1,000 a month in infrastructure. Self-hosted Portkey plus their control plane lands in between with much better governance than anything you would build yourself.
Teams underestimate the ops line every time. Running LiteLLM well in production is more work than the README implies.
Migrating off raw SDKs
Less scary than it looks. Every gateway here speaks OpenAI-compatible chat completions, so the mechanical change is the base URL and the key.
The real work is the routing config: fallback chains, timeout budget, retry policy, cache rules. Decide those before you cut traffic over. Run in shadow mode for a week, duplicate traffic, compare outputs, watch the cost dashboard, then flip.
The part nobody warns you about is the urge to keep adding gateway features once you have one. Start with routing and fallbacks. Add cost attribution when finance asks. Add guardrails when legal asks. Not all of it on day one.
If you have been putting this off, point your smallest production agent at OpenRouter or Portkey for an afternoon and look at a week of cost data. That is usually the moment the gateway stops being optional.