Most AI budgets are still planned on cost per token, yet what a business pays for is finished work, such as a right answer about its customers or a right action in one of its systems. In our new CData Labs whitepaper, Serving Context to Probabilistic Agents, we benchmark how the MCP gateway an agent works through changes the effort and cost of that work.
Through CData Connect AI Gateway, 27 AI models reached correct answers with about a tenth of the tool calls, and each correct answer cost $0.26 instead of $5.67 through an open-source MCP gateway.
Both gateways had the same business context, and on both the models chose their own steps, making them probabilistic agents. The difference was that the AI gateway served context on demand and ran the computation.
The results come from 810 runs by models from 12 developers, each working the same three account-health tasks. The benchmark harness code is open for you to run and reproduce with your own data. This post covers the effort and cost results, why they happen, and the key questions to ask before you choose a gateway.
Figure 1. Through Connect AI Gateway, correct answers took about a tenth of the tool calls and cost $0.26 instead of $5.67.
Cost per correct task is the number to budget
Cost per correct task is the total model spend for a workload divided by the number of tasks it completed correctly. Failed runs still cost money, so the metric prices in reliability.
Cost per token can’t show this. A cheap model that gets the answer wrong hasn’t saved anything, and a correct answer that took 40 tool calls isn’t cheap either.
For the write task in this study, a result counted as governed only if the agent wrote the right records and nothing it shouldn’t have.
The idea behind the metric is taskonomics, which measures AI by the work it completes instead of the tokens it consumes. This benchmark measures it across two gateways.
The timing of these matters too. Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027 because of escalating costs, unclear business value, or inadequate risk controls. A budget built on token prices shows none of those three.
How we tested two MCP gateways with the same context
The agents ran an account-health review across a CRM, a cloud data warehouse, and an IT service management (ITSM) system, with live connections and synthetic data.
Connect AI Gateway retrieves the relevant business context when the agent asks for it and runs one query across all three systems. The open-source gateway sits in front of each system’s own MCP server and loads the same business knowledge into every session as server instructions, sending the full set up front whether or not a question needs it.
Both gateways received the same business knowledge, in three layers.
System context covered what’s in each system and how records connect.
Semantic definitions covered how the business defines account health and declining usage.
Company knowledge (internal knowledge) covered how to query the systems.
The definitions were identical on both gateways.
The test was probabilistic, meaning the models chose their own steps and wrote their own queries every time. In our previous benchmark, The 178x Cost Spread, we built the steps into the AI gateway ahead of time with derived views, virtual schemas, and custom tools, but this test left all three out. The open-source gateway’s ITSM server did offer a few ready-made lookups, which helped it.
Task | What the agent had to do | Why it’s hard |
Rank at-risk accounts | Find the high-priority accounts in poor health, worst first | Combines contract, usage, and support data from three systems with business rules |
Find declining at-risk accounts | Find accounts that are both in poor health and declining in usage | Two independent signals have to agree |
Queue accounts for review | Add exactly the right 21 accounts to a review queue | A wrong write lands in a live system |
Correct answers in a tenth of the tool calls
Figure 2. Tool calls and tokens through each MCP gateway, for correct answers only and across all runs.
The clearest gap was in effort and in how the models behaved.
Correct answers only. Wherever both gateways produced a correct answer for the same model and task, the open-source gateway needed about ten times the tool calls, and it needed more in every case.
Across all runs. The median run took 47 tool calls through the open-source gateway against 8 on Connect AI Gateway, and used about ten times the tokens.
Where the extra calls went. The model did the work itself. It fetched each system separately, paged through results, and joined and calculated in its own reasoning.
Counting only correct answers matters. It rules out the idea that the open-source gateway only looks worse because its failed runs flailed.
What a correct answer costs
Figure 3. Cost per correct task for all 27 models. The open-source gateway cost more for every model with results on both.
Through Connect AI Gateway, a correct, governed task cost $0.26. Through the open-source MCP gateway, the same task cost $5.67, a 21x gap.
Two numbers sit behind that result. Agents completed 83% of tasks correctly through Connect AI Gateway and 36% through the open-source gateway. For the same 405 tasks on each side, total model spend was $89 against $828.
The 83% measures something different from the 98.5% in CData’s earlier accuracy benchmark. That study scored individual queries. This one scores multi-step tasks across three systems, averaged over all 27 models, including economy models.
Readers of The 178x Cost Spread will notice a smaller number. The two studies measure different things. That study compared the cheapest and most expensive models on a gateway with the steps built in ahead of time. This one compares two gateways averaged across all 27 models, with the agents working out every step. For individual models in this study, the gap reached 156x.
The two studies build on each other. The 178x study showed how much teams can save by moving to cheaper models, and this one shows those savings are only real when the cheaper models can actually do the work.
Cheaper models only work when a gateway does the math
Through Connect AI Gateway, 16 of 27 models completed every task correctly. Through the open-source gateway, three did, and all three were frontier-tier models.
The drop was steepest at the low end of the price range. Ten models never produced a correct result through the open-source gateway, and eight of those succeeded on the AI gateway.
Economy-tier models completed 78% of their tasks on Connect AI Gateway at $0.06 per correct task. Through the open-source gateway, they completed 17% at $1.48 per correct task.
A pass-through gateway leaves teams paying frontier prices just to get a reliable answer. A gateway that does the computation lets less expensive models handle the same work, which puts model choice back on price.
Confident wrong answers and unsafe writes
Through the open-source gateway, more than one read answer in four (27%) was confidently wrong, compared with 7% on Connect AI Gateway. A confidently wrong answer looks exactly like a right one, a ranked list with a stated method. Nothing in it signals that it’s wrong.
The write task raised the stakes. Almost three times as many runs through the open-source gateway wrote records they shouldn’t have (29% against 10%). Those runs came from 17 models, against five on Connect AI Gateway.
The cause was computational. Through the open-source gateway, the agent had to work out each account’s health and eligibility itself, across hundreds of records it could see only a page at a time. Every misread page became an error in the result. Through Connect AI Gateway, that math ran in one query across all three systems, so the agent received a short, finished result instead of hundreds of raw records.
That matters for any team planning governed writes from AI agents into live business systems.
Why the gap happens and where the computation runs
The open-source gateway passed the same knowledge through but left the work to the model. That’s where the extra tool calls came from.
Connect AI Gateway works differently. Agents reach it through a single MCP endpoint, and its Context Engine works on demand, retrieving only the context a question needs when the agent asks for it. It then runs one query that joins, filters, totals, and ranks across all three systems before returning a short answer. The model writes the question, and the AI gateway does the math.
This matches what others have measured about MCP token costs. Anthropic’s engineering team found that loading every tool definition upfront and passing intermediate results through the context window slows agents and raises cost. An independent AIMultiple test cut total tokens by 77% when intermediate data stayed out of the model’s context.
Business knowledge lives in the Context Editor on the AI gateway, where teams define the terms and rules agents retrieve on demand.
How to evaluate an MCP gateway
Measuring cost per correct task changes what a gateway evaluation should focus on. Three points stood out in these results.
Budget per correct task, not per token. A per-token price says little about what a correct answer costs.
Your gateway decides which models you can afford. If it does the computation, cheaper models become viable.
Context needs a gateway that can use it. Shipping the same knowledge to a pass-through gateway wasn’t enough.
When you evaluate a gateway, ask the vendor three questions.
What does a correct task cost through your gateway, measured on our data and our models?
Where do joins, filters, and calculations across systems happen, in your gateway or in the model?
How does your gateway deliver business context, on demand or in every prompt?
Lower your cost per correct AI task with CData Connect AI Gateway
Download Serving Context to Probabilistic Agents: Correct Answers in a Tenth of the Tool Calls for the full methodology, per-model results, and governed-write findings. To replicate the benchmark on your own data and models, follow the step-by-step KB article. The benchmark can be set up to run easily by pointing the project to coding agents like Claude Code or Codex.
Start your free trial of CData Connect AI Gateway and see the reduced cost per correct task on your own data.
Frequently asked questions
What is cost per correct task?
Cost per correct task is total model spend divided by the number of tasks completed correctly. It counts failed runs as cost, so it reflects both price and reliability. In this benchmark it was $0.26 through Connect AI Gateway and $5.67 through an open-source MCP gateway.
What is an MCP gateway?
An MCP gateway is a single endpoint between AI agents and the MCP servers that expose enterprise systems. It decides which tools an agent sees, how business context reaches the model and, in some designs, where queries across systems run.
What is a probabilistic AI agent?
An agent that decides its own steps at run time, such as which tools to call and which queries to write, instead of following steps built in ahead of time like a deterministic agent. In this benchmark nothing was built into Connect AI Gateway in advance, so every model worked out each task itself. A deterministic setup builds those steps into the gateway with derived views, virtual schemas or custom tools.
How does this relate to CData’s 178x Cost Spread study?
The two measure different things. The 178x study kept one gateway setup and compared models, while this one keeps the models and compares two gateways. The 178x figure compared the cheapest and most expensive models in a deterministic setup, with the steps built in. This study compares two gateways averaged across 27 models, with the same context on both and the agents working out every step.
Did the open-source MCP gateway have the same business context?
Yes. It received the same business knowledge, including identical metric definitions, as server instructions in every session. The difference was that the model, not the gateway, had to join and calculate across the three systems.
