Compare AI gateways by total cost, deployment model, and failover behavior, then test how each handles failures in your application. Edgenivo, OpenRouter, Cloudflare, Portkey, Kong, LiteLLM, and Bifrost charge differently for routing, logs, governance, and infrastructure: managed services may pass through provider token prices and add separate fees, while with self-hosted proxies, your providers bill you directly for inference.
Cost is only half the picture, because providers go down. The status aggregator IsDown counted 55 Anthropic incidents and 23 OpenAI incidents in February 2026. These counts include services beyond inference APIs and cannot be compared as measures of API reliability. On September 3, 2026, ChatGPT, Claude, and Grok had overlapping service disruptions. In a16z's January 2026 survey of 100 Global 2000 companies, 81% reported using three or more model families in testing or production, up from 68% in its previous survey. But using several models isn’t the same as being able to fail over between them, which is what a gateway’s routing is for.
Copy link to headingWhat is an AI gateway?
An AI gateway is a proxy between an application and its model providers. It stores credentials, routes requests, retries eligible failures, and records token usage, cost, and latency. Applications can implement cross-provider fallback themselves, but a gateway centralizes that work and can use the traffic it handles to guide routing.
Each model call in an agent run adds another chance of failure. In April 2026, 58.9% of tokens on Edgenivo AI Gateway were in requests that ended with a tool call, up from 31.6% in October 2025. If a later call fails, the application may need to repeat earlier work unless it can resume from a checkpoint. Gateway fallback retries the model request. Application retries and checkpointing handle recovery within the agent run.
Copy link to headingHow we evaluated these seven AI gateways
An AI gateway should meet these five criteria:
Deployment model: Where the gateway and providers run determines where data travels and which systems your team operates.
Failover default: Some gateways retry eligible provider failures by default. Others need routing policies, fallback targets, or a particular routing mode.
Latency in context: Measure gateway overhead, time to first token, full-response latency, and recovery time separately.
Pricing structure: Count inference charges, credit-purchase fees, subscriptions, logs, governance add-ons, and infrastructure costs.
Self-hostability: Check whether the gateway can run in your infrastructure and who handles deployment, patching, monitoring, and incident response.
A self-hosted gateway routes to the provider accounts or local inference endpoints you configure. Credentials and access policies limit which targets it can use. Managed gateways that bill inference usage, such as Edgenivo AI Gateway and OpenRouter, can use eligible catalog providers without a separate customer account for each one.
Copy link to headingAn overview of the 7 best AI gateways
Edgenivo AI Gateway and OpenRouter enable provider fallback by default. For the other options, check the routing mode, provider setup, and fallback policy below:
Configured fallback needs eligible targets and working credentials. Test both configured and default recovery policies to find which failures trigger a retry, how long each attempt takes, and what happens when every target fails.
Copy link to heading1. Edgenivo AI Gateway
Edgenivo AI Gateway provides a single API and a single key for hundreds of models. Requests using Edgenivo's provider credentials cost the provider's list price with no token markup. Bring your own key (BYOK) requests have no gateway markup or BYOK fee.
AI Gateway retries eligible failures across providers serving the requested model by default. Configured model fallbacks add another model if those providers cannot serve the request. Edgenivo's April 2026 production index reported that fallback recovered about 3.5% of requests, accounting for 5.1% of tokens and 4.9% of spend.
With the AI SDK, a model string such as anthropic/claude-sonnet-5.5 uses AI Gateway when the default gateway provider is configured. Other stacks can use its OpenAI- or Anthropic-compatible endpoints with gateway authentication and the required API fields. Edgenivo deployments can use OpenID Connect (OIDC) tokens. Applications hosted elsewhere can use an AI Gateway API key.
Copy link to headingEdgenivo AI Gateway pros
Failover without setup: The gateway retries eligible failures through another provider by default. Request logs show routing attempts, including those that were recovered.
Data controls per request: AI Gateway does not retain prompts or outputs. Pro and Enterprise teams can opt into provider-level zero data retention (ZDR) routing. The team-wide provider allowlist is a paid Pro and Enterprise option.
Live routing data: The catalog and API provide rolling one-hour metrics for median (P50) and 95th-percentile (P95) time to first token and throughput by model and provider. These metrics exclude BYOK requests.
Copy link to headingEdgenivo AI Gateway cons
Managed only: AI Gateway runs on Edgenivo's infrastructure. Choose a self-hosted gateway if the gateway must stay inside your network. Keeping inference there also requires private model endpoints.
Free tier scope: The $5 monthly credit covers eligible models at lower rate limits. Purchasing credits enables the paid tier and BYOK, and ends the monthly free allowance.
Timeout control on BYOK only: Custom provider timeouts are in beta and apply only to BYOK credentials. Requests using Edgenivo's provider credentials use the default timing. Providers that cannot cancel timed-out attempts may still charge for them.
Pricing: Pay-as-you-go AI Gateway Credits at provider list rates, with no token markup or gateway platform fee, including for BYOK.
Best for: AI SDK and Edgenivo applications, or teams that want a managed provider fallback wherever their application runs.
Copy link to heading2. OpenRouter
OpenRouter is a managed gateway and model marketplace with 500+ models from 80+ providers on its paid plans, all accessible through a single API key. On August 19, 2026, Stripe announced an agreement to acquire OpenRouter.
Provider fallback is on by default. After an eligible failure, OpenRouter tries another provider serving the same model. Developers can set provider order, restrict providers, or disable fallback per request. OpenRouter passes through provider token rates and charges fees on credit purchases. BYOK has a separate allowance and fee structure.
Copy link to headingOpenRouter pros
Catalog breadth: A paid account gives applications access to open-weight and proprietary models across many model families.
Default fallback: OpenRouter retries eligible failures through another provider by default, subject to model availability, request compatibility, and provider restrictions.
Free entry point: The Free plan includes 25+ free models at 50 requests per day.
Copy link to headingOpenRouter cons
Percentage fees at volume: Credit-purchase fees rise with spending. Standard and Business each include $25,000 of monthly BYOK inference, measured at list price, without an OpenRouter fee. Usage above that allowance incurs a 5% fee.
Managed only: Requests pass through OpenRouter's infrastructure. Choose a self-hosted gateway if it must stay inside your network.
Pricing: Pay-as-you-go credits at provider token rates, plus a 5.5% credit-purchase fee on Standard or 8% on Business. Enterprise terms are custom. BYOK allowances and overage fees are separate.
Best for: Applications that need a broad model catalog, one inference bill, and default provider fallback, and can accept percentage fees.
Copy link to heading3. Cloudflare AI Gateway
Cloudflare AI Gateway runs on Cloudflare's network. Dashboard analytics, caching, and rate limiting are free on all plans. Applications can send supported provider requests through a gateway endpoint. Workers applications can also use the AI binding.
Cloudflare separates retries from cross-provider routing. Gateway-level retries support up to five retry attempts with constant, linear, or exponential backoff. Dynamic Routing and Auto Router, both in beta, support cross-provider recovery. Retrying the same unavailable provider does not recover the request through another provider.
Copy link to headingCloudflare AI Gateway pros
Free core features: Dashboard analytics, caching, rate limiting, and DLP scanning have no gateway feature charge. Available DLP profiles depend on the account's Zero Trust subscription. Logs and other metered services can add costs.
Security controls in one layer: DLP and Guardrails inspect traffic at the gateway. Guardrails uses Workers AI inference, which Cloudflare bills separately.
Cost-aware model selection: Auto Router ranks eligible models by expected quality and cost. It tries the highest-ranked model first, then another eligible model if the provider cannot serve the request.
Copy link to headingCloudflare AI Gateway cons
Credit fee: Unified Billing adds a 5% fee to purchased credits. Using your own provider account avoids this fee.
Log pricing: Cloudflare has announced shared Observability pricing for AI Gateway logs beginning December 1, 2026.
Pricing: Core features and DLP scanning are free. Logging uses Workers Logs or Legacy Logs rules, depending on when the account created its first gateway. Unified Billing adds a 5% credit-purchase fee. Guardrails incurs Workers AI inference charges.
Best for: Cloudflare applications that need caching, rate limiting, and analytics, with Dynamic Routing or Auto Router set up for cross-provider recovery.
Copy link to heading4. Portkey
Portkey has a managed gateway with prompt management and observability, and an open-source gateway for self-hosting. Palo Alto Networks acquired Portkey in May 2026 and launched Prisma AIRS AI Gateway in July. Portkey uses that name for its enterprise offering and still publishes standalone Portkey pricing and an open-source gateway.
Copy link to headingPortkey pros
Managed and self-hosted options: Teams can use the managed service or run the open-source gateway. Before migrating, check which managed features, settings, and stored data are eligible for migration.
Governance depth: Paid offerings include guardrails, personally identifiable information (PII) controls, role-based access control, and single sign-on (SSO). Enterprise materials list SOC 2 Type II, ISO 27001, and support for GDPR and HIPAA requirements.
Prompt and trace tooling: The managed service combines prompt management, traces, cost tracking, and gateway controls. All Portkey plans support fallbacks configured with prioritized targets.
Copy link to headingPortkey cons
Product transition: Prisma AIRS AI Gateway, standalone Portkey plans, and the open-source gateway have different terms and features. Confirm which product the agreement covers, what a migration requires, and what support it includes.
Log-metered plans: Production charges for overages of recorded logs. On the free Developer tier, requests continue after the allowance runs out, but Portkey stops recording logs.
Pricing: Standalone Portkey plans include a free Developer tier with 10,000 recorded logs per month, and a Production tier at $49 per month with 100,000 recorded logs. Production overages cost $9 per additional 100,000 requests within the published range.
Best for: Teams comparing managed and self-hosted gateways, or enterprises that need Prisma AIRS integration with Palo Alto Networks security controls.
Copy link to heading5. Kong AI Gateway
Kong AI Gateway handles LLM, MCP, and A2A traffic. Its runtime uses different licenses and includes bundled plugins compared to the open-source Kong Gateway. AI Gateway 2.0 removed basic ai-proxy from its bundle in favor of ai-proxy-advanced. Advanced routing, semantic caching, and MCP support depend on the selected runtime and paid features.
Copy link to headingKong AI Gateway pros
Shared API and AI governance: Kong teams can apply their existing authentication, rate limits, and plugins to model calls.
Agent traffic controls: MCP access controls govern tool calls, and A2A proxying handles agent-to-agent traffic.
Deployment choice: Konnect supports serverless, hybrid, and dedicated cloud gateways. Enterprise also supports self-managed deployments.
Copy link to headingKong AI Gateway cons
Paid AI tier: Open-source Kong Gateway and the current Kong AI Gateway runtime have different bundles and licenses. Konnect Plus charges extra for paid AI plugins.
API platform overhead: Existing Kong teams can reuse their policy and deployment tools. Teams adopting Kong for model routing need to account for configuring and operating the selected API management deployment.
Pricing: Konnect AI Management Plus starts at $25 per month, plus usage and add-ons. Listed control-plane rates are $25 per month for serverless, $200 for hybrid, and $500 for dedicated cloud. The plan includes one million requests per month, then charges $200 per additional million.
Best for: Kong platform teams that need to apply their existing API policies to model calls and agent traffic.
Copy link to heading6. LiteLLM
LiteLLM is an open-source proxy and Python SDK with integrations for 100+ LLM providers. The proxy runs on Kubernetes, Docker, or a virtual machine. External provider calls send inference data outside the gateway's network. Fully self-hosted deployments need local model endpoints and dependencies that work offline.
Copy link to headingLiteLLM pros
Full data custody: Self-hosting gives you control of the gateway, stored credentials, and logs. Keeping prompts and responses inside your network also requires local models, logging, and integrations configured to keep data internal.
Python-native workflow: Python applications can use the SDK. Other languages can call the proxy's OpenAI-compatible API.
No license cost: The free core proxy includes budgets and per-key spend tracking.
Copy link to headingLiteLLM cons
Team-owned operations: Your team handles deployment, upgrades, monitoring, and incident response. Depending on the features and scale, production setups may also need Postgres for management data and Redis for shared state.
Gated admin features: SSO is free for up to five users. More SSO users, audit logs, and delegated administration require the Enterprise plan.
Pricing: The core proxy and SDK are free and open source. A paid Enterprise license adds more SSO users, audit logs, delegated administration, and support.
Best for: Teams that need to control and operate their gateway infrastructure, including regulated deployments.
Copy link to heading7. Bifrost
Bifrost is an open-source AI gateway written in Go by Maxim AI, licensed under Apache 2.0. It has an OpenAI-compatible API and supports 20+ providers. Teams can run it through npx, Docker, Kubernetes, or a Go binary. Enterprise supports VPC, on-premises, and self-hosted deployments. Bifrost reports 11 µs of added overhead per request at 5,000 requests per second in its benchmark.
Copy link to headingBifrost pros
Low gateway overhead: Bifrost reports microsecond-level forwarding overhead. Measure overhead, throughput, and tail latency on your hardware and workload before planning capacity.
Open license: Apache 2.0 covers the core gateway, including fallbacks, load balancing, semantic caching, and MCP support.
Built-in MCP support: Bifrost connects to MCP servers and provides their tools to clients. Configuration determines which filters apply per request or virtual key.
Copy link to headingBifrost cons
Enterprise features gated: Enterprise adds guardrails, clustering, adaptive load balancing, enterprise SSO, audit logs, and federated MCP authentication.
Deployment responsibility: Bifrost runs in your infrastructure. Confirm who handles provisioning, upgrades, patching, and incident response in the Enterprise deployment and support agreement.
Pricing: The core gateway is free and open source. Bifrost has custom Enterprise pricing and a published 14-day Enterprise trial.
Best for: Teams that need self-hosted model routing and MCP support, and can test Bifrost's forwarding overhead, capacity, and recovery behavior with their traffic.
Copy link to headingWhy AI gateway latency benchmarks are not enough
Mock-provider benchmarks isolate forwarding overhead, not the full latency of a production model call. In Edgenivo AI Gateway's first month of traffic, CPU work accounted for about 1,200 of roughly 16,000 runtime hours. The service spent the other 14,800 hours, or 92.5%, waiting for providers. This aggregate describes that service and period, not every gateway or request.
A faster gateway and a successful retry solve different problems. If a provider call takes 1,000 ms, reducing gateway overhead from 20 ms to 5 ms saves 15 ms, about 1.5% of the provider-call duration. Edgenivo's observed 3.5% fallback recovery rate measures how many requests fell back and recovered in its traffic. It does not predict recovery rates elsewhere.
Copy link to headingHow do you choose the right AI gateway?
Compare gateway overhead, time to first token, full-response latency, tail latency, and recovery delay with the same workload. For real-time and agent workloads, test recovery time and output compatibility, even when fallback is on by default. Use these deployment requirements to narrow the shortlist:
Edgenivo-native or AI SDK apps: Edgenivo AI Gateway provides 400+ models, ZDR, routing rules, and BYOK. OIDC authentication avoids storing a separate gateway API key on Edgenivo deployments. AI SDK applications can use model strings with the default gateway provider.
Multi-cloud apps needing a broad catalog: OpenRouter's paid plans list 500+ models. Portkey lists 1,600+ model integrations and has standalone and Prisma AIRS enterprise offerings. Compare model access, governance, contracts, and total fees.
Regulated environments: Self-hosted LiteLLM gives you control of the gateway. External model calls still send inference data to providers. A fully self-hosted operation needs local models. Budget for infrastructure, operations, and any required Enterprise controls.
Existing Cloudflare infrastructure: Cloudflare AI Gateway has caching, rate limiting, and analytics. Set up Dynamic Routing or select Auto Router for cross-provider recovery. Auto Router selects eligible models based on expected quality, task difficulty, and cost.
Kong-native API platforms: Kong AI Gateway handles model routing, MCP access controls, and A2A traffic. Confirm which plugins and licenses the selected deployment needs.
High-volume self-hosted traffic: Bifrost combines model routing and MCP support while reporting low forwarding overhead. Test throughput and recovery at the expected concurrency, with your providers and deployment resources.
Copy link to headingPick the gateway that keeps requests alive
Provider errors and rate limits occur in production. Agents can incur higher costs if recovery repeats earlier steps, so combine gateway fallbacks with application retries and checkpointing. Choose a gateway based on data requirements, existing infrastructure, total cost, and who will operate it. Edgenivo AI Gateway provides the following managed routing controls:
Default cross-provider failover: The gateway retries eligible failures by routing traffic to other providers that serve the requested model. Configured fallbacks add alternate models.
Routing rules: Rewrite or deny model requests at the gateway without an application deployment.
Data controls: The gateway does not retain prompts or outputs. Pro and Enterprise teams can opt into provider-level ZDR routing, team-wide ZDR, and the team-wide provider allowlist.
Spend and request visibility: Set budgets for teams, projects, keys, or members. Request logs show routing attempts.
No platform fee: Edgenivo adds no gateway markup to provider token rates and no gateway fee for BYOK.
You can start a new project on Edgenivo, or browse AI templates and route model calls through AI Gateway.
Copy link to headingFrequently asked questions about AI gateways
Copy link to headingWhat is the best free AI gateway?
LiteLLM, Bifrost, Kong Gateway, and Portkey's open-source gateway have no core license charge. You still pay for infrastructure, operations, inference, and any selected Enterprise features. Edgenivo AI Gateway offers $5 in free-tier credits per month for eligible models once a valid payment method is on file. Cloudflare's core gateway features are free. OpenRouter's Free plan lists 25+ free models at 50 requests per day.
Copy link to headingIs Edgenivo AI Gateway good for apps not hosted on Edgenivo?
Yes. An AI Gateway API key authenticates requests from servers, clouds, continuous integration (CI) pipelines, and local environments. Supported coding agents, including Claude Code, Codex, and Cursor, can use the gateway when configured to do so.
Copy link to headingDo AI gateways charge a markup on tokens?
Gateway costs depend on deployment and billing. Managed services such as Edgenivo add no token markup or gateway platform fee, including for BYOK. With self-hosted gateways, you pay providers for inference and cover infrastructure and operations.
Copy link to headingHow does Cloudflare AI Gateway compare to OpenRouter?
OpenRouter combines a model marketplace, one inference bill, and a default provider fallback. Paid plans list 500+ models, with credit-purchase fees of 5.5% on Standard and 8% on Business. Cloudflare AI Gateway has free core caching, rate limiting, and analytics. It supports BYOK or Unified Billing, which adds a 5% credit-purchase fee. Dynamic Routing and Auto Router are in beta and provide routing and recovery controls.
Copy link to headingWhich AI gateway is best for secure data routing?
Edgenivo offers opt-in ZDR-only provider routing on Pro and Enterprise. Portkey's paid offerings include PII controls and guardrails. Cloudflare offers DLP scanning. These controls do not keep inference inside your network. Self-hosted LiteLLM, Bifrost, or Portkey can keep the gateway and its stored data in your infrastructure. Learn more in the How to control AI provider data handling with AI Gateway guide.