Stop Making Agents Guess: Engineering Context in the Gateway Path

At AgentCon in October, Amit Naik, CData's vice president of AI architecture, walked through the life of a prompt. Three agents in the same company, working on the same data, got the same question: "What enterprise revenue is at risk this quarter?" They returned three confident answers: $6.8 million, $2.1 million, and $4.9 million. None of them had the context that defines "enterprise," "revenue," "this quarter," or "at risk" for that business.

If you've been following our recent posts, you know that story. The final third of the talk took on a harder concept. Most teams now agree that context is critical for agent accuracy, so the open question is architectural: should the context layer sit beside the AI gateway, or inside its request path? Where it sits decides what it can learn.

Sprawled context doesn't stick

Many teams have already given their agents context of some kind, but the trouble is keeping it. The same four problems come up again and again:

  • Agents forget what they learn between sessions. An analyst tells an agent on Tuesday that bookings exclude professional services. On Wednesday, the agent starts over without that correction.

  • What one user teaches an agent doesn't reach the next user. A correction lives in one person's conversation history. The next person who asks the same question gets the same wrong join.

  • Skills are files, not governed context. Every user and agent keeps its own copy. Nothing ensures they're all running the approved version, and an update reaches only the copies someone remembers to replace.

  • Skills are static. A skill records what someone knew on the day they wrote it. When the fiscal calendar shifts or a new escalation policy ships, the skill stays out of date until someone edits it by hand.

context chaos the key problem

Problems cascade as context is lost

When context doesn't persist, the costs add up across the whole organization, not just in one agent.

Inconsistent answers. The three at-risk numbers aren't a corner case. They're what happens by default when each agent, model, and user carries its own idea of what a term means. Once two dashboards disagree, people stop trusting either one.

Answers no one can audit. When context lives in scattered skill files and memory stores, there's no record of which definition produced a number, who approved it, or which sources it touched. Sharing raises the risk: a memory or skill that captures what one user could see can surface that data to an agent working for someone who couldn't. Governance that stops at the tool call doesn't cover what's inside the context.

Time spent reteaching. Someone has to repeat every correction an agent forgets, often the same expert, over and over. The domain knowledge that should build up instead gets typed back in.

Context windows full of bloat. To make up for agents that forget, teams load every schema, definition, and skill into the prompt just in case. Tokens go to exploring rather than answering. In CData Labs' internal benchmark, resolving context before the model runs cut input tokens by more than 90% compared with loading the full catalog into the prompt. It also replaced a dozen schema-crawling round trips with a single call.

Context scattered across tools. When definitions live in skill files, prompts, and memory stores spread across tools and teams, sending a request to a different model or agent means hoping the same context is waiting there. That undercuts one of the main reasons to have a gateway: routing each request to the best model for the job. In CData's September 2026 cost study, models that all answered correctly, once the data layer did the work, varied 178x in cost per correct answer. Teams can only capture that spread if their context moves with them.

Workarounds act as temporary band-aids

Teams are trying four approaches. Each one fixes part of the problem and creates a new one.

Large agent skill libraries. The first instinct is to write down everything the agent needs. Libraries grow to hundreds of skills, and nobody owns all of them. Some contradict each other. Most go stale. The skill format now works across tools, but each tool loads skills from its own folder, and nothing tracks which version is approved or who's using it. Skills exist client side, not server side.

Putting everything in a data lake. If all the data lives in one place, the reasoning goes, the context can live there too. But moving every source into a lake is expensive, and replicated data trails the source by however long the pipeline takes. Federation and zero-copy sharing narrow that gap for reads. The harder limit is action: agent workflows that update a renewal date or open an escalation have to write back to the source systems, and the lake doesn't own those.

Standardizing on one model. If context only works reliably in one runtime, picking one provider looks like the simple fix. In practice, it raises switching costs, gives away negotiating power, and gives up the cost and accuracy gains of routing each request to the best model.

Adding an agent memory service. Cloud providers now offer long-term memory that persists across sessions and can be shared between agents. That helps an agent recall what was said. It doesn't tell the agent what "revenue" means, how accounts join to contracts, or what a given user is allowed to see. And each memory service is tied to one cloud's runtime.

context inside path

Put context inside the gateway path

All four workarounds put context in a separate system and connect it to agents afterward. Amit argued that this placement is the root problem.

Much of the industry is building context as a catalog, semantic layer, or context repository that sits beside the gateway. The gateway sends it a lookup and gets a static answer back. Unless a team builds a feedback loop around it, it doesn't see the data that came back or how the user reacted, so it has little to learn from. Once it routes a request, it forgets it.

Placement also decides whether context is enforced or just available. If context is something an agent can consult, it's also something an agent can skip. Give the same agent a raw SQL tool and it can query around the approved metric definitions. Context in the request path applies to every request, whether or not the agent asks for it.

The alternative is to put the context layer inside the gateway's request path, where intent meets execution.

Beside the gateway

Inside the gateway

What it sees

Only the lookup it's been asked

The prompt, the returned data, and the user's reaction

How context arrives

A static answer to a query

Injected into the conversation after a real-time policy check

What happens after

Starts all over, for every user and every interaction

Validated answers improve the next request

Data platforms are moving in this direction too. Lakehouse and warehouse vendors now register models, agents, and Model Context Protocol (MCP) servers in their catalogs and govern them from one layer. For questions about data that already lives in that platform, that can be enough. But that context stops at the platform's edge. It doesn't reach the sales, finance, and support systems whose data never landed in the warehouse, and it ties agents to that vendor's models and tools. The at-risk question needs Salesforce, NetSuite, and ServiceNow at once, and most enterprises run more than one data platform.

CData Connect AI takes the inside-the-path approach. Its Context Engine sits in the AI gateway alongside the LLM Gateway, the MCP Gateway, and a virtual data layer over live sources. Because every agent, user, and model request passes through that one point, context isn't tied to any single one of them:

  • Shared across agents and users. Finance approves a revenue definition once, and every agent that asks about revenue gets the same one. A correction one analyst makes is available to everyone.

  • Independent of the model. Context is resolved in the gateway before the request reaches a model. You can route to whichever model fits the job and take the 178x cost spread without re-creating your definitions.

  • Governed and traceable. Every definition carries an approval and lineage back to its source. Queries run under the requesting user's identity, so each source system enforces its own permissions, and shared context doesn't widen what anyone can see.

  • Live, with no lake. The virtual data layer runs federated queries against Salesforce, NetSuite, ServiceNow, and other sources where they live. It filters and joins at the source, returns only the final dataset, and supports actions that write back to those systems.

  • Built to learn. This is where the architecture is headed: once a human validates an answer, the Context Engine saves its context and the calculation behind it as a reusable skill. The gateway injects that skill before the model has to guess next time, so the trial and error happens once, for everyone. CData is building this now with early-access customers.

gateway flow

A fair question is whether this just moves the lock-in into CData's gateway. The difference is what the gateway holds. The data stays in your source systems, and you choose the model per request. Connect AI works across data platforms and is built to consume the semantic definitions your teams already maintain, not replace them. If you change models or add a warehouse, your context comes along.

Resolving schema and permissions before the model runs also moves accuracy. Across 378 real-world prompts and five MCP approaches, schema-aware, governed tools answered correctly 98.5% of the time. The other approaches scored between 65% and 75%. At 75% accuracy per step, a five-step agent workflow finishes correctly less than a quarter of the time. These studies measure what resolved, governed context does for an answer. Placement decides whether every request gets that context, and whether it improves over time.

Putting context in the path has costs. It takes upfront work to deploy and maintain, and every request passes through one more layer, which can add latency depending on the architecture. In return, one resolved call replaces the dozen round trips an agent makes when it has to explore the schema on its own.

Ask three questions of your gateway

Amit ended the talk with three questions to ask any AI gateway, CData's included:

  1. See: Does it see the data that came back, or only the prompt?

  2. Know: Does it know what "revenue" means in your company, or does the model guess?

  3. Remember: Does it remember what worked last time, or start from zero?

If the answer to any of these is no, the gateway can route requests, but it can't improve them. Context earns its place only where it sees every request, every result, and every correction. You can run the open accuracy harness against whatever gateway you use today, and read the full accuracy and cost studies. Early access to CData Connect AI Gateway is open now.

Explore CData Connect AI today

See how Connect AI excels at streamlining AI and business processes for real-time insights and action.

Get the trial