As AI spreads across the enterprise, the pressure to reduce LLM token costs is getting harder to ignore, and the reason is counterintuitive: the bill keeps climbing even as token prices fall. Lower unit prices haven't solved the cost problem, because agentic AI asks models to do more work, across more systems, through longer chains of reasoning and tool calls. In fact, Gartner predicts AI inference cost per agentic workflow will rise more than fivefold through 2028.
So how should you get ahead of these rapidly rising costs? Most of the focus today is centered around the model layer, as teams look to use cheaper models, prompt caching, and compression. Those help, but some of the largest under-leveraged savings sit upstream, in how much data and tooling the model has to take in before it can answer. The tokens you never send are the cheapest of all. Controlling what reaches the model is the job of an AI gateway, and specifically of its data-access layer, the part that sits between your agents and your enterprise data and shapes the request before it ever hits the model. Most gateways stop at routing model traffic and never reach that layer.
Our token savings calculator is the fastest way to get a handle on your own spend: enter your number of AI users, expected usage, and model, and it estimates your context-related token spend today and what it could look like optimized. It only takes about a minute to run.
CData Connect AI is the gateway built to cut token spend at the data layer. With Connect AI, you can federate across your sources, filter and pre-shape data before it reaches the model, and expose purpose-built tools instead of raw schemas. In our open, replicable benchmark, a fully optimized Connect AI setup cut token use by as much as 97.6% on a live, multi-source query.
Below, we'll break down which parts of a request consume tokens, why agentic and multi-source workflows multiply the cost, and how you can cut that spend at the data layer.
Where your tokens actually go
When someone asks what one AI request really costs, the visible prompt is only part of the answer.
Token overhead is the share of a request's tokens spent on tool bloat and response bloat, rather than on reasoning or the answer itself. In a basic chatbot exchange, that overhead stays small. Connect an agent to enterprise systems through Model Context Protocol (MCP), and both grow fast.
Tool bloat comes first. Before an agent can call a tool, the model has to be told the tool exists, so it reads tool names, descriptions, parameters, and schemas loaded into context up front, before any useful work begins. Anthropic has published an example in which 58 tools across five MCP servers consumed roughly 55,000 tokens before the conversation even started.
Response bloat comes after the tools are called. A CRM request might return dozens of fields and thousands of rows when the answer needs five fields and 20 records. That result lands in the context window, where the model reads it, interprets it, and often carries parts of it into later tool calls.
MCP often takes the blame for this, but MCP is only the transport protocol. The overhead comes from how the client, the application hosting the model, loads tool definitions and raw results into context, not from the protocol itself. In enterprise workflows, tool bloat and response bloat can dominate the request.
Why agentic and multi-source workflows multiply the cost
The problem sharpens as AI moves from answering questions to completing business tasks. Consider a question like: which enterprise customers have open support tickets, no renewal in the next 90 days, and below-threshold product usage?
The answer spans several systems: a customer relationship management (CRM) system for accounts, an IT service management (ITSM) tool for support cases, a warehouse for usage telemetry, and another system for contract and renewal data. An agent using a conventional discovery pattern has to identify the systems, inspect schemas, find the relevant objects, retrieve field definitions, run separate requests, move each result into context, work out how the records relate, and only then synthesize the answer.
Every additional system adds another schema, another tool call, another result set, and another inference. Each round-trip is another turn, and on every turn the model re-reads all the context built up so far. As a request runs more turns, its token consumption compounds.
Wider adoption multiplies that cost rather than amortizing it. A workflow that looks cheap in a five-person pilot behaves differently when dozens of employees run it repeatedly, agents execute it automatically, and every run reconstructs the same discovery chain. That is the hidden token tax: paying again and again for the model to rediscover and reread what the architecture could have narrowed beforehand.
Observe after the fact, or reduce at the source
Most AI cost control today starts with visibility and limits. Usage dashboards show which teams consume the most tokens, budget alerts fire when spending crosses a threshold, and rate limits cap how much an agent can consume. Model routing and prompt caching go further and genuinely lower cost, routing simpler work to cheaper models and reusing repeated context. Each of these has a place.
But they work around the overhead rather than removing it. Visibility tells you a workflow spent 180,000 tokens; those tokens still went through the model. A rate limit controls whether a request runs, not what it costs when it does. And caching and routing, useful as they are, don't touch the large schemas and raw multi-source results that create much of the overhead in the first place.
A more direct form of token optimization asks a different question: how much of this context can be removed before the request reaches the model? Anthropic's own guidance on agent tooling recommends returning only high-signal information and using filtering, pagination, range selection, and truncation so tool responses don't bloat the context. That is tool bloat and response bloat addressed at the source. For enterprise data, you can push it one layer deeper and do more of the data work outside the model entirely.
The architecture that cuts the model's workload
An AI gateway is a centralized control layer that sits between AI consumers (your applications, agents, and copilots) and the models, tools, and data systems they use. Model calls, tool invocations, and data queries all flow through it. Most evaluations focus on the model-routing side, but the layer that governs which enterprise data reaches the model is where tool bloat and response bloat are created, and where they can be removed.
CData Connect AI is that data-access layer: an enterprise-grade, managed MCP platform reaching hundreds of sources through one governed endpoint. With Connect AI, you can federate queries across systems, push filters down to each source, shape results, and expose purpose-built tools, so the model receives a smaller, finished answer instead of orchestrating raw calls. Connect AI takes on the data work the model would otherwise do in-context, so the model reasons over the finished result rather than your entire data model. You configure token efficiency once, and every AI workflow reuses it:
Custom Tools and Toolkits let you go the other way: define a parameterized SQL tool that returns only the fields and filters a workflow needs, then bundle those tools behind one scoped MCP endpoint, so the agent discovers a short, purpose-built list instead of the whole catalog.
Derived Views pre-join, filter, and transform data across sources into reusable virtual tables, so the model sees a finished result rather than orchestrating three calls itself.
Workspaces limit an agent to an authorized, relevant catalog instead of the full schema.
Jobs and caching serve recurring queries from prepared data instead of hitting the live source again.
These features also cut the number of turns a request takes. When a scoped tool hands back a finished answer, the model stops looping through discovery. Fewer turns mean less context to reprocess on each pass, which lowers the token cost.
The key distinction is where the optimization happens. Dashboards measure waste after the model processes it, and routing to cheaper models only makes that waste cheaper. Connect AI reduces the material before the model runs, and because that logic is reusable, your team sets up an efficient data path once and applies it across every workflow, instead of asking each agent to rediscover the same architecture.
What the savings look like in practice
To measure just how much an optimized Connect AI configuration could save in token costs, we ran an open, replicable benchmark on a live multi-source question spanning Salesforce, Snowflake, and ServiceNow.
On the raw, baseline run we were comparing against, the agent took the fully exploratory path, relying on Universal Tools (the standard built-in operations Connect AI exposes for any connected source) to discover connections, schemas, objects, and fields. That run consumed 183,541 tokens across 22 tool calls, at about $0.596 per request. Exposed through a Custom Tool built for the multi-source question, the same request used 4,427 tokens in a single call, at about $0.027. That is a 97.6% reduction, the measured best case when a Custom Tool is well-aligned to the query.
Less context didn't mean a worse answer. The optimized path returned the same result from a fraction of the context. Narrowing the tool and data surface stripped irrelevant information while preserving what the question required. The precise economics vary by model, workflow shape, request volume, and the features deployed, but the principle travels: the model should reason over the data the answer needs, not everything it could reach.
Estimate your own token savings
A 97.6% benchmark reduction is useful evidence, but the more important question is what token optimization means at your own scale. The CData Connect AI token optimization calculator lets you model your own token spend using your number of AI users, expected usage, and model.
The calculator uses each model provider's published pricing and an assumed reduction in context processing. Actual savings depend on your workflow shape, model, request frequency, data volume, and the features you deploy. Run your own inputs today and bring the output to our team for a workflow-specific analysis.
Reduce the cost of context, not just the price of tokens with Connect AI
Lower model prices help, but enterprise AI cost increasingly hinges on something upstream: how much work the model is asked to do. Every unnecessary schema, raw row, repeated discovery call, and intermediate result competes for space in the context window and adds cost at scale.
Token optimization shifts the focus from monitoring those costs to engineering them out of the workflow. Start a free trial of CData Connect AI to configure that efficiency on your own stack, or book a call with our team to map the savings to your specific workloads today.
Frequently asked questions
Why do MCP servers use so many tokens?
MCP itself doesn't consume context. Token usage comes from how MCP clients expose tools and move results through the model. Tool catalogs and schemas are loaded into context so the model knows what it can call, and tool results are inserted afterward so it can reason over them. Large catalogs and verbose responses can consume tens of thousands of tokens before the final answer is generated.
Does reducing tokens hurt answer quality?
Not when optimization removes irrelevant context instead of required information. A smaller, well-filtered result reduces noise and makes the relevant data easier to reason over. CData's benchmark recorded a 97.6% token reduction while the optimized workflow also produced more accurate responses.
How is this different from usage dashboards or rate limits?
Dashboards measure consumption after it happens, and rate limits cap how much is allowed. Neither changes how many tokens a single request needs. Connect AI targets the request itself by narrowing tools, federating data, pushing down filters, pre-joining sources, and caching appropriate results before the model runs.
How much could my team save?
It depends on your model, number of users, request frequency, workflow complexity, and the features deployed. The CData Connect AI token optimization calculator estimates your before-and-after spend from those inputs. Run your own assumptions to see a number for your environment.
