If you are reading this, assume you already have an artificial intelligence (AI) gateway in place. Your model traffic is routed, your costs are controlled, and if you have followed this series, you may have model context protocol (MCP) governance for agent-to-tool calls as well. The traffic layer is handled. And yet the AI is still returning answers that are subtly wrong, adoption is plateauing, or the security review keeps asking a question the gateway cannot answer: how does the model know it is allowed to see this data? Routing gateways govern how AI requests are handled. They do not govern what the AI knows. A gateway with no connection to live enterprise data moves the problem downstream, where the request is routed efficiently and the answer is still wrong.
Speaking at the 2026 Gartner Data and Analytics Summit, analyst Andres Garcia-Rodeja predicted that 60% of agentic analytics projects relying solely on MCP will fail by 2028 for lack of a consistent semantic layer underneath. MCP moves context. It does not produce it. This post covers what routing gateways are built to do, why routing cannot fix correctness, what context means for enterprise AI, the three kinds of context an enterprise depends on, how context and routing fit together, and what to look for when you add the context layer.
This is the fifth post in the CData executive learning series on the AI gateway.
Read the full series: The AI Gateway: An Executive Learning Series
At a glance
Definition: An AI gateway context layer is the part of an AI stack that supplies a model with accurate, current, permission-filtered data from enterprise systems of record. Routing decides how a request is handled; the context layer decides what the model knows.
The gap: A routing gateway optimizes the path a request takes but never touches the systems that hold the answer. Without a context layer, well-routed requests still return stale, incomplete, or over-permissioned answers.
Why it matters: AI errors and hallucinations are the top-cited barrier to enterprise adoption, and most are data context failures that routing cannot fix.
What routing gateways are built to do
Before naming the limitation, it is worth crediting what routing gateways solve, because they solve it well. A routing gateway centralizes model selection, enforces rate limits and cost controls, propagates user identity through tool calls, logs traffic for audit, and applies security policy at the request layer. These are real infrastructure problems, and the LLM gateway and MCP gateway covered earlier in this series handle them properly. Nothing that follows is an argument against having one.
The point is architectural scope. A routing gateway sits between your applications and your models or tools. It sees the request and the response, and it optimizes the path between them. What it does not touch is the enterprise systems that hold the data the AI is being asked about, because those systems sit upstream of the router entirely. Consider an agent asked for the current open pipeline. The gateway routes that request to the right model efficiently. But where the model gets the pipeline figures, how current they are, whether they are filtered to what this particular user is allowed to see, and whether "pipeline" even means what the user assumes it means, are not routing questions. The routing layer is complete at the traffic layer. The question this post addresses is what has to exist at the data layer for the traffic layer to return answers that are correct, trustworthy, and compliant.
The correctness problem routing cannot solve
The failure mode has a precise shape: a request routed to the best model at the best price still produces a wrong answer when the data the model receives is incomplete, stale, or scoped beyond what the user is authorized to see. Routing is blind to all three conditions, because none of them are visible at the traffic layer. Call it the blind router problem. A gateway with no connection to enterprise data systems routes requests without any understanding of what context those requests require, and a well-routed wrong answer is still a wrong answer.
The downstream cost of that gap is already measurable. Gallagher's 2026 survey of more than 1,200 businesses found that AI errors and hallucinations top the list of perceived adoption threats, cited by 57% of respondents, ahead of legal and reputational risk. Newsweek, reporting industry estimates the same year, put the annual cost of AI hallucinations in the tens of billions of dollars. These are not primarily model failures. More often they are data context failures, and the routing layer cannot address them no matter how well it is tuned.
In practice the correctness problem shows up in three recognizable forms. In the first, the AI has no access to the relevant data at all, so it guesses or declines. In the second, it receives a stale copy from a periodic index rather than the live system, and answers confidently from information that is already out of date. In the third, it receives more data than the requesting user is authorized to see, which is a governance failure whether or not the answer is correct. Each one requires a layer that routing does not provide.
What "context" means for enterprise AI
Context, for an enterprise AI application, is the accurate, current, and permission-filtered information a model needs to answer a question correctly, drawn from the systems of record that hold the actual state of the business. That definition is worth stating plainly, because the word gets used loosely and the enterprise version is harder than the consumer one. Consumer AI grounds its answers in public information that changes slowly. Enterprise AI has to ground answers in live operational data, the customer relationship management (CRM) record updated this morning, the inventory count from the last warehouse sync, the headcount as of today, all of it sitting in access-controlled systems with no public interface built for AI consumption.
It also helps to say what context is not. It is not a system prompt, a prompt template, or a document index. Those are mechanisms for delivering context, not context itself. A retrieval-augmented generation (RAG) index that refreshes nightly is better than nothing, but it is a periodic snapshot, and most deployed AI applications are working from snapshots they do not know have gone stale. Real enterprise context has three parts: the live data in systems of record, the semantic definitions that give that data business meaning, and the tacit knowledge that shapes how an organization reads both. The first two are addressable with current technology. The third is where the field is still heading.
The three kinds of enterprise context
The first layer is live company data. These are the operational records in enterprise systems, queried in real time rather than copied in advance. An application working from training data or a nightly index is always answering from a version of the business that has already moved on, and for anything time-sensitive, current pipeline, latest inventory, this quarter’s headcount, that lag produces wrong answers delivered with full confidence. The fix is live query access to the systems of record, not more frequent ingestion. The market is discovering this in real time: VentureBeat's Q1 2026 tracking showed enterprise intent to adopt hybrid retrieval, which combines live access with indexed retrieval, tripling from 10.3% to 33.3% in a single quarter as index-only approaches hit their limits.
The second layer is semantic definitions. This is the business meaning behind the fields, the relationships between objects across systems, and the agreed definitions that turn raw schema into something interpretable. An AI that can query live Salesforce data but does not know that "Amount" on an opportunity refers to projected closed value, rather than total contract value or annual contract value, will misinterpret data it retrieved correctly. Semantic definitions translate schema into meaning, and they are the layer that tools like dbt, LookML, and warehouse-native semantic models already address for analytics. A context layer has to carry the same definitions for AI to reason correctly across systems.
The third layer is tacit knowledge: the implicit vocabulary and reasoning shortcuts an organization’s people use to interpret their own data. How a specific sales team defines a qualified lead, which revenue line a business unit reports to, the exception rules that exist in practice but in no system of record. This is the hardest layer to formalize and remains an active area of development across the industry, so treat it as the direction the category is moving rather than a solved problem. Even the first two layers are a significant lift for most organizations. The connection to correctness is what matters here. Live data without semantic definitions gets misread, and live data and definitions without permission filtering expose records the user should never see. Each layer is necessary, none is sufficient alone, and correctness lives at the intersection of all three.
How context and routing work together
None of this makes a context layer a replacement for a routing gateway. The two operate at different points in the request lifecycle and solve different problems, and a complete AI infrastructure needs both. Routing governs which model handles a request and at what cost. Context governs what that model knows when it runs. Neither substitutes for the other.
Inside a single request, the two work in sequence and then together. The routing layer selects the model. The context engine determines what data to retrieve, from which systems, filtered to the requesting user’s permissions, and enriched with the semantic definitions that make it interpretable. Both decisions resolve before the model runs, and the model receives the routing decision and the context at the same time. There is a cost relationship worth noticing here as well. A context engine that returns only the fields and rows a query needs, already filtered by permission, shrinks the token payload the model has to process, which lowers inference cost regardless of which model routing selected. Context quality and routing efficiency compound: fewer tokens mean lower cost on any model. The architectural conclusion is straightforward. The LLM gateway and MCP gateway are the traffic half of governed AI. A context engine is the data half. Either half alone leaves a gap; both together produce AI that is governed, correct, and cost-efficient.
What to look for when adding a context layer
If you are evaluating this layer, whether you plan to build it or buy it, the same questions apply, so it is worth framing them as evaluation criteria rather than a feature checklist. Does the layer query live systems of record, or does it serve from a periodic index that can go stale? Does it enforce the requesting user's own permissions at the point of data access, before any data reaches the model, so the agent acts as the user rather than above the user? Does it carry real semantic definitions, business terms, cross-object relationships, and agreed field meanings, rather than raw schema metadata alone? Does it cover the breadth of sources your actual workflows touch, including custom fields and on-premises systems? And does it produce per-query logs detailed enough for compliance and cost attribution?
The build-versus-buy decision follows from how broad your data footprint is. An organization with narrow, stable requirements and dedicated platform engineering can reasonably address this layer with a focused internal build. An organization with many SaaS systems, on-premises databases, and custom applications usually finds that the integration and maintenance burden of building it exceeds the cost of adopting something purpose-built. The honest question is less build or buy than what building actually requires at your scale. At CData, we are building a context layer inside Connect AI designed around these criteria, combining data source context, cross-source relationships, and intelligent context retrieval. The next post in this series takes these evaluation criteria apart in detail.
Learn what to look for when evaluating a context layer
The next post in this series covers evaluation criteria for context layer in detail: source coverage, schema intelligence, semantic resolution, and permission enforcement at the data layer. CData Connect AI is building this layer for enterprise AI, giving applications and agents governed, real-time access to hundreds of enterprise data sources with each user’s permissions enforced before data reaches the model. To go deeper on the requirements, read 7 essential data requirements for agentic AI, or see how the data access layer fits alongside your model routing.
Frequently asked questions
Why does an AI gateway without data context give wrong answers?
An AI gateway governs how requests are routed, optimizing model selection, cost, and security policy at the traffic layer. It does not govern what data the model receives. Without a context layer connected to live enterprise systems, the AI works from training data or a periodic index that may be incomplete, stale, or scoped beyond the requesting user's permissions. Hallucinations on enterprise queries are usually not model failures but data context failures. A routing gateway can deliver those wrong answers with excellent efficiency; only a context layer can prevent them.
What is the difference between a context layer and a RAG pipeline?
RAG retrieves documents from a periodic index based on semantic similarity, and it works well for knowledge bases, documentation, and unstructured content. A context layer addresses the additional demands of operational data: live query access to transactional systems instead of periodic indexes, permission enforcement at retrieval time, and structured resolution across multiple systems. The two are complementary rather than competing, RAG for unstructured content and a context engine for live operational records, and most production enterprise AI deployments eventually need both.
How can AI agents inherit user-level permissions when accessing enterprise systems?
The pattern is called delegated authorization, or identity pass-through. The agent authenticates on behalf of the requesting user, and every data query it makes runs under that user's credentials against the source system. The source system's own access controls, Salesforce sharing rules, SAP authorization objects, and database row-level security, then apply to the agent exactly as they would to that person querying directly. The context layer is the right place to implement this, because it is the layer that actually makes the data queries. The routing layer never touches the source systems.
How does a context layer reduce AI token costs?
Without a context layer, an agent requesting enterprise data often receives the full schema and unfiltered records for the model to sort through, spending input tokens on data the query never needed. A schema-aware context engine returns only the fields and rows relevant to the specific query, already filtered to the requesting user's permissions, so the reduction happens at the source before the model runs rather than relying on the model to discard the excess afterward. Field-level filtering of this kind commonly cuts payloads by a wide margin. In CData's own benchmark, a scoped tool definition cut token usage by 97.6% on a multi-source enterprise query spanning Salesforce, Snowflake, and ServiceNow, from roughly 184,000 tokens to 4,400 across 56 runs.
Explore CData Connect AI today
See how Connect AI excels at streamlining AI and business processes for real-time insights and action.
