A sales leader types a standard question into their chat: What revenue is at risk for the mid-market segment in Q3?
This prompt reads like a question a model should handle easily. It’s a question that’s asked almost every week across every company. But a model can’t reliably reason through this on its own. Four pieces need to be settled so that a model can answer it correctly: the company's definition of mid-market, the fiscal calendar that turns Q3 into a real date range, which of several revenue figures counts, and what anyone here means by at risk.
Those answers live in Salesforce, NetSuite, ServiceNow, a data model built two years ago, and the judgment of a support team. Hand the question to a model cold and the answer comes back fluent, specific, and wrong.
Here is what the CData Connect AI Gateway does with that prompt in the moment before an answer comes back.
A surprisingly complex question
Before heading directly to the model, the prompt is intercepted by the gateway.
A model that gets this question directly will certainly answer it, because answering is what models do. It will pick a plausible definition of mid-market, assume a calendar quarter, choose whichever revenue field it finds first, and infer what at risk means from the shape of the data. Every one of those choices is a guess, and none of them is visible in the answer.
The Connect AI gateway removes the guessing by settling the question before inference starts.
Intent gets classified before anything runs
The first thing the gateway decides is what kind of question this is: a lookup, an analytical question, or a workflow. This one is a lookup. It wants a value that already exists somewhere, not a projection or a chain of actions.
That classification picks the model. An unresolved question defaults to the largest reasoning model available, because the model has to do the interpretive work itself: figure out what the terms mean, reconcile the sources, decide what counts. A resolved question arrives with all of this figured out. Once the ambiguity is gone, a smaller and cheaper model returns the same answer.
In typical setups, model selection gets treated as a tuning decision made in advance, per use case. It's better understood as a consequence of how much context the model is handed. By resolving more upstream, the routing gets cheaper on its own.
Routing is configured, not improvised. In the LLM Gateway you define a routing rule and attach it to a project. The rule is an ordered list of provider-and-model pairs, each carrying its estimated cost per thousand tokens, plus a rule type that says how to move down the list: priority failover sends traffic to the first model and falls through to the next when the first is unavailable. A rule can be set as the workspace default so every project inherits it.
Clients never hold a provider key. They authenticate to the gateway with a virtual key scoped to a project, which is what makes the routing swappable: change the rule and the next request lands on a different model without a line of application code changing. The guardrails and spend limits attached to that same project apply whichever model ends up serving the request.
A routing rule in the LLM Gateway: an ordered model list with estimated cost per thousand tokens, so routing is a policy the gateway applies rather than a choice made in application code.
"At risk" is not a trusted column in any database
The Connect AI Gateway’s context engine reads the prompt against a context graph, that is organized by the kind of knowledge each term needs.
The agent calls a search_context tool with the question in plain language and gets back the slice of the catalog that matches: the datasets, metrics, and glossary entries behind those terms, each with its description, synonyms, semantic type, fields, and join hints, plus the fully qualified name to query it by. One call returns only the connections that caller is authorized to use, with their permissions already attached.
Mid-market and Q3 resolve against system context, the segmentation rules and fiscal calendar the business already runs on. Q3 comes from the accounting periods defined in NetSuite, so a fiscal year that opens in February resolves to August through October rather than the calendar months a model would assume. Mid-market comes from the derived view over Salesforce accounts that encodes the band finance and sales agreed to, so it is a formal account list rather than a threshold the model picked.
Revenue resolves against the semantic layer. Most enterprises have several fields called revenue: booked, recognized, billed, ARR. The semantic layer holds which one this organization means in this context, so the question doesn't quietly answer itself with the wrong field that an analyst takes up to the board.
At risk is the hardest one, and the most useful to look at closely. No system has an at-risk column. The signal lives in tacit knowledge: support ticket patterns, SOPs, the things a renewals team knows and has written down somewhere. This is the category most AI data layers skip, because it isn't structured and doesn't join cleanly. It's also where the actual business meaning sits.
Resolve those four terms and the question stops being ambiguous, and nothing downstream has to infer meaning.
With shared context, global concepts sit at the top, and the entities that feed them are drawn from the systems and data models underneath.
It's worth opening one of those concepts, because this is where the tacit knowledge becomes literal.
The Account Health entry includes synonyms so the term resolves however someone phrases it. It adds a definition assembled from three signal families across Salesforce, Zendesk, and Snowflake, and then a data-quality warning: the Total_Health_Score__c field looks like a ready-made answer, but it's populated on 1 of 131 accounts, so it should be treated as an unfinished feature rather than a source of truth.
That last finding here represents an oft missed flag for teams. This type of data quality fault is what an analyst learns the hard way and then carries the learning in their head. Written into the context graph, it becomes something every model inherits, closing the trap that would have produced a confidently wrong answer before anyone hits it.
Choosing the sources, binding the tools
With the terms resolved, the model knows which systems it needs to reach: Salesforce for accounts and pipeline, NetSuite for financials, ServiceNow for support signal, and the derived view that holds the segment definition. The model does the choosing, and it calls the tools itself.
What the gateway decides is what is on the menu. Twelve tools are available on this connection; the gateway scopes the set to the four this task needs and the model calls within that set. The model stays free to reason about which tool fits, while what it can reach at all stays a property of the connection rather than a matter of prompt discipline.
This combats tool bloat. A common failure in agent deployments is handing the model every tool on every call, which inflates the context window, raises latency, and widens the blast radius of a mistake. Selecting tools per task keeps all three down, and the selection is a function of the resolved question rather than a configuration someone has to maintain by hand.
With Connect AI’s MCP gateway, each tool server carries a trust ceiling, so what an agent is allowed to reach is set on the server rather than negotiated per request.
Policy travels with the connection
Each connection carries its own permissions and policies:
Source | Permission |
Salesforce | Read/write, selected fields |
NetSuite | Read-only, one report |
ServiceNow | Read, PII masked |
Derived view | Read-only |
First, the policy is enforced at the connection, not in the prompt. Instructing a model not to look at something is not a control, it is a request, and it often fails discreetly when the model has the data in its context window. Here the data the policy excludes never enters the context window at all.
Second, the policy is set per identity. The same prompt from a different person or agent pulls a different result set, because permissions resolve against who is asking. This is the condition many AI deployments fail on when they move into production: the demo runs with one service account that can read everything, and the production version has to respect the access model the company already has.
Workspaces are how the scope gets drawn. account-health and account-health-write expose the same subject matter at different permission levels, so an agent is handed the one that matches what it's entitled to do, and no part of the other is reachable from it.
The join happens outside of the model
Cross-source joins rest on the connectivity layer CData has been building for twenty years. Every connected system is presented as a relational catalog behind one SQL interface, whatever it speaks underneath, whether that is a REST API, SOAP, or a proprietary ERP endpoint. Salesforce accounts and NetSuite transactions are therefore queryable the same way and joinable in the same statement, and filters and aggregations push down to each source where that source can run them.
The query engine lives in the gateway, not just as a proxy, so the join can happen before the model sees anything.
Permitted records move into the gateway, masked where the policy says so, and the gateway does the work that is usually left to the model.
Salesforce and NetSuite records join on the derived view's segment definition. ServiceNow's at-risk flags append as a column, so what was four result sets in four schemas becomes one comprehensive table.
A derived view is a saved query that behaves like a table. Someone defines mid-market once, as the join and the filters and the band that make up the segment, names it, and it lives in the CData catalog as [CData].[DerivedViews].[Mid_Market_Accounts], queryable like any other table. The definition is maintained in one place instead of being re-derived in every prompt, dashboard, and pipeline, which is what makes two people asking the same question get the same number.
The typical, alternative approach is to pass all four result sets into the context window and let the model reconcile them. That works in a demo and degrades in production, because the model is doing a deterministic join probabilistically. It has no way to know that an account id in Salesforce and a customer record in NetSuite are the same company unless something tells it, and when it gets that wrong, the answer is still articulate.
200,000 tokens become 9,500
What reaches the model is one resolved, verifiably accurate table.
Passing the raw result sets would have cost roughly 200,000 tokens. The resolved table costs about 9,500. This represents a 95% reduction, and this saving is a beneficial side effect rather than the primary goal. The gateway isn't compressing the payload. It's sending only what the question actually needs, which turns out to be a small fraction of what was retrieved.
Three things follow cost reduction from that:
A model given one table with resolved terms has nothing left to guess at.
Less to read means a faster response, and a smaller model can serve it.
Context that isn't spent on raw records is available for the conversation.
The answer returns through one more policy check before it reaches the user, so what came back can't carry anything the asker wasn't entitled to see.
The gateway keeps logs and learnings
Two learnings persist after the answer is delivered.
The audit log. Every step is recorded: the prompt, each action taken, each permission validated, each record accessed, the response returned. It’s not just a log that an answer was produced, but a log of what the system touched to produce it. When someone asks six months from now why a number looked the way it did, or when an auditor asks what that model had access to, the record exists. Reconstructing this after the fact is not possible, which is why it has to be captured at the point of execution.
It is also one log rather than several. In most other setups, an LLM gateway sits in front of the models and point each source at its own native MCP server. This approach splits the record in two: the gateway knows which model ran and what it cost, each native server knows what was read out of its own system, and nothing holds the line between them. Answering what a given model touched then means pulling logs from every source and reconciling them by timestamp, assuming each one logged enough to reconcile.
Connect AI sits on both sides of the exchange, the model traffic and the data access, so the prompt, the permission check, the records read, and the response are rows in the same log. One query answers what happened, across every source and every model.
Query log and audit log are accessible separately. Every row carries a timestamp, the user who triggered it, the connection it touched, and the statement that ran.
Agentic memory. The resolution encodes back into the context graph. The next question involving mid-market, or Q3, or at-risk accounts, starts from what was worked out here instead of resolving it again.
This memory is portable across models. It belongs to the gateway, not to whichever model happened to run or data lake contains the data. Switch from one provider to another, one data source to the next, and the accumulated context comes along. This also means the usual lock-in argument doesn't apply, because the asset is the context layer rather than the model.
Now ask Connect AI your hardest questions
One sentence in. One accurate answer back. 95% fewer tokens, policy enforced down to the record, a complete audit trail, and a context graph that knows more than it did a minute ago.
None of that came from a better model. It came from resolving access, structure, and meaning before the model ran.
With CData Connect AI, your team can ask questions against live enterprise data without building the resolution layer first, which is the layer most AI projects discover they need only after the pilot works and production doesn't.
See Connect AI against your own data
