In early 2026, Uber burned through its entire annual budget for artificial intelligence (AI) in four months, after encouraging its engineers to use AI coding tools as much as possible and ranking their usage on internal leaderboards (reported by TechCrunch). Every one of those dollars went to ordinary engineering work, prompts tested, internal tools built, agentic workflows run, each request metered by the token and subject to no ceiling and no attribution, until the finance team found the year’s budget already gone. In April. Uber's public overrun is quiet story for many others.
In most organizations, each application hardcodes a single frontier model, sends every request to it regardless of how simple the task is, and offers no visibility into what the model is doing once it reaches production. The large language model (LLM) gateway is the layer that brings that traffic under control. It sits between your applications and the model providers they call, and it decides which model handles each request, what that request costs, and whether the call needed to happen at all. This post defines the LLM gateway, separates it from the API gateway and the AI gateway it keeps getting confused with, and shows where it belongs in your AI stack.
This is the fourth post in the CData executive learning series on the AI gateway, running every Tuesday and Thursday through mid-September.
Read the full series: The AI Gateway: An Executive Learning Series
At a glance
Definition: An LLM gateway is an infrastructure layer that sits between an application and one or more model providers, exposing a single API while centralizing multi-model routing, cost tracking, semantic caching, observability, and failover.
Where it sits: The LLM gateway governs model traffic. It nests inside the broader AI gateway, which also governs tool calls and data access, and it runs alongside the MCP gateway that governs agent-to-tool traffic.
Why now: Enterprise model spend is doubling in half-year steps, and the model layer is the first place most organizations feel the cost, reliability, and lock-in pressure a gateway is built to absorb.
What an LLM gateway is
An LLM gateway is an infrastructure layer that sits between an application and one or more model providers, exposing a single API while centralizing multi-model routing, cost tracking, semantic caching, observability, and failover. That single interface is the reason it exists. Instead of every team maintaining separate software development kits (SDKs), authentication setups, and billing accounts for each provider, they integrate once with the gateway, and switching a model or adding a fallback becomes a configuration change rather than a code change.
Four functions define a proper LLM gateway. It handles routing and failover, deciding which model serves each request and where traffic goes when a provider degrades. It handles cost tracking and enforcement, attributing every token to a team or project and capping spend before it runs away. It provides observability, logging routing decisions, latency, and token usage for each call. And it applies security guardrails, screening requests before they reach a provider. The deeper benefit is decoupling: when your application talks to one stable interface rather than a specific provider’s SDK, the choice of model becomes a runtime decision instead of an architectural commitment you have to rebuild around later.
LLM gateway vs API gateway
It is reasonable to ask whether an API gateway already covers this. Application programming interface (API) gateways are mature, well-understood infrastructure: they manage authentication, rate limiting, versioning, and lifecycle management for REST services, and they were built for synchronous, fixed-cost HTTP calls. Model traffic breaks several of those assumptions at once. Requests are billed by the token rather than priced per call, responses stream back incrementally rather than returning as a single payload, the threat model includes prompt injection rather than only authentication bypass, and useful caching depends on the meaning of a request rather than an exact URL match.
The cost gap is the clearest example. An API gateway can tell you how many requests hit an endpoint. An LLM gateway tells you how many tokens each request consumed, what it cost by model, which team generated the spend, and whether a semantic cache hit could have avoided the call altogether. None of that is a defect in API gateway design; it reflects what the technology was built to do. The two are complementary: use an API gateway for your service and microservices layer, and use an LLM gateway for your model access layer. Most enterprises running production AI operate both. We covered the parallel case for agent-to-tool traffic in the MCP gateway post earlier in this series.
LLM gateway vs AI gateway: where the boundary is
The terms "LLM gateway" and "AI gateway" are used inconsistently across vendors and analysts, and the labels alone will not settle which one you are looking at. Some vendors use them interchangeably to mean model-routing middleware. Others reserve "AI gateway" for a broader control plane that also governs agentic tool invocations and MCP access. The cleanest way to hold them apart is by scope. An LLM gateway is model-side infrastructure: it routes traffic between applications and providers, optimizes cost, and provides observability at the model-call layer. An AI gateway in the broader sense wraps that and adds governance for tool calls, made through the model context protocol (MCP).
The practical implication follows directly. For teams whose applications call model APIs directly, such as chatbots, summarizers, and code assistants, an LLM gateway is the right primary control layer. For teams deploying agents that reach enterprise tools and data, an LLM gateway is necessary but not sufficient, because a data access layer is also required. That distinction, model routing on one side and governed data access on the other, sets up the question of where each piece belongs in the stack.
How model routing works
Model routing is the mechanism behind most of an LLM gateway’s cost savings. Rather than sending every request to the same frontier model, the gateway examines each incoming request and directs it to the model best suited to the task, based on complexity, cost, latency, or policy. A query that needs multi-step reasoning goes to a frontier model, and a simple lookup or classification routes to a smaller, cheaper one. The savings come from a fact most teams underestimate, which is that a large share of production traffic never needed a frontier model in the first place.
The research backs this up with specific numbers. RouteLLM, an open framework from LMSYS published at ICLR 2025, trained routers on public preference data and found that its best router held 95% of GPT-4 performance on the MT Bench benchmark while sending as few as 14% to 26% of requests to the expensive model, a cost reduction of roughly 75% to 85% on that workload against a random baseline. The exact figures depend on the benchmark and the model pairing, so treat them as proof the technique works rather than a number you will hit unchanged. Semantic caching compounds the effect. When a new prompt is close enough in meaning to a recent one, the gateway returns the stored answer and skips the model call entirely, and the gateway is the right place to implement it because it already intercepts every request before it reaches a provider.
Observability, fallback, and redundancy
Cost control is only half of why an LLM gateway earns its place in production. The other half is operational reliability, and it starts with observability. Every request through the gateway is logged with its routing decision, latency, token counts, and cost, which turns "what is the model doing" from an investigation into a query. This is the diagnostic layer teams usually retrofit after their first production incident, and building it in from the start is what keeps that incident from being a surprise.
Failover is the reliability pattern that matters most. A gateway configured with more than one provider can automatically reroute to a secondary model when the primary is unavailable or exceeds a latency threshold, which converts a single-provider outage that could take a feature offline into a routing event users never notice. Enterprises with their own graphics processing unit (GPU) infrastructure often run a related pattern, routing routine requests to on-premises models and falling back to cloud providers only when local capacity runs out. Sitting above all of this is budget enforcement. Because every token flows through one place, the gateway can alert before spend reaches a threshold, cap consumption per team or project, and block requests from a runaway workflow, turning an unpredictable cost line into a managed one.
Where the LLM gateway fits in the complete AI stack
It helps to picture a production AI deployment as three layers. The application layer is where AI features live. The model layer is where the models live, governed by the LLM gateway. The data layer is where your enterprise data lives, governed, secured, and auditable. Most teams reason carefully about the first two layers and underbuild the third, and that third layer is where a large share of stalled AI pilots trace back to.
An LLM gateway does not close that gap, and it was never meant to. It routes traffic to models efficiently, but it does not decide how a model gets authorized access to current data behind an answer, whether that data lives in a CRM, an ERP, an HR system, or another system of record. That problem lives at the data layer. At CData, we build Connect AI as the data access platform that completes the stack. CData Connect AI gives AI applications and agents governed, real-time access to hundreds of enterprise data sources through a single MCP-compliant interface, so the data reaching a model has been permission-filtered, schema-optimized, and audit-logged before the model ever sees it. Where an LLM gateway manages which model handles a query, Connect AI manages what data that model is allowed to read and write. Together they form the full control plane for production enterprise AI: an LLM gateway governing the model layer, a data access platform governing the data layer, and your applications on top.
Give your LLM governed access to live enterprise data with CData Connect AI
An LLM gateway gives your organization one place to route, meter, and keep model traffic observable. The harder half of the problem, giving those models governed access to the live data behind every answer, is where CData Connect AI does its work. Connect AI provides governed, real-time access to hundreds of enterprise data sources, from Salesforce and SAP to Snowflake and ServiceNow, through a single MCP-compliant interface. Route your model traffic through your LLM gateway, and route your data access through Connect AI. Start a free trial and connect your first source in minutes, or read the documentation to see how the data-access layer fits alongside your model routing.
Frequently asked questions
What is an LLM gateway?
An LLM gateway is an infrastructure layer that sits between an application and one or more model providers, exposing a single API while centralizing multi-model routing, cost tracking, semantic caching, observability, and failover. Instead of integrating each provider's SDK separately and managing multiple authentication setups, developers point existing code at the gateway endpoint and switch models by changing a single parameter. The gateway handles routing decisions, cost attribution, and reliability behind the scenes.
How does model routing work in an LLM gateway?
The gateway examines each incoming request and routes it to the most appropriate model based on complexity, cost, latency, or policy. Simple queries such as summaries, lookups, and classification route to cheaper, faster models, while complex reasoning tasks route to frontier models. Research from RouteLLM (LMSYS, ICLR 2025) found that a well-trained router held 95% of GPT-4 performance on the MT Bench benchmark while sending only 14% to 26% of requests to the expensive model, a cost reduction of roughly 75% to 85% on that workload.
What is the difference between an LLM gateway and an AI gateway?
"LLM gateway" and "AI gateway" are used interchangeably by some vendors and distinctly by others. The clearest distinction is scope: an LLM gateway is model-side infrastructure that routes traffic between applications and providers, optimizes cost, and provides observability at the model-call layer. An AI gateway in the broader sense also governs tool invocations, data access, and agentic workflows. For teams deploying agents that reach enterprise data, an LLM gateway is necessary but not sufficient, because a data-access governance layer is also required.
Where does an LLM gateway fit in the AI stack?
The LLM gateway sits in the model layer, between applications and model providers. A complete AI stack has three layers: the application layer where AI features run, the model layer where models are governed by the LLM gateway, and the data layer where enterprise data is governed by a data-access gateway like CData Connect AI. Most teams invest heavily in the application and model layers, and the data layer is the most commonly underbuilt, which is a primary cause of AI pilot failure.
Explore CData Connect AI today
See how Connect AI excels at streamlining AI and business processes for real-time insights and action.
