Why Enterprise AI Is Over Budget and Under-Delivering

Welcome to the CData back-to-school AI executive learning series, where over the next month, across eight parts releasing every Tuesday and Thursday, we'll work through one of the industry's newest architectural concepts from the ground up: the AI Gateway. We'll cover why it emerged, what it's designed to solve, where it still falls short, and how your enterprise can fold it into a broader AI infrastructure strategy. We kick off today with the root cause behind the demand for it: why so much enterprise AI is running over budget and under-delivering.

Enterprise AI was supposed to follow a predictable path. Pilot projects would demonstrate value, measured rollout would follow, and ROI would justify the investment. What most organizations are finding instead is that production AI is wildly expensive, difficult to forecast, and in most cases, completely ungoverned. McKinsey's Enterprise AI FinOps Survey (May 2026, 75 respondents across five major industries) found that 93% of organizations are already exceeding their AI budgets, with spend increasing nearly fourfold as deployments scale from isolated use cases to enterprise-wide. Sixty-two percent of those organizations have already moved past experimentation. The challenge isn't getting AI to production. It's that production AI is generating invoices nobody fully anticipated, and most organizations won't understand why until the next procurement cycle.

In this article, we look at three structural gaps driving enterprise AI cost overruns, and the architectural changes that address them.

The AI budget crisis is a production problem, not a pilot problem

Most enterprises have reached a specific moment: AI is live, your agents are running, and the first full procurement cycle has closed. What leadership bought as a promising infrastructure investment is generating invoices that nobody fully anticipated, for reasons that remain opaque to the teams being asked to explain them.

Part of what makes this hard is that AI pricing doesn't behave like any prior enterprise software category. Seat licenses are predictable. Consumption is not. Token usage can vary by up to 30x executing the same task across different workflows, prompting strategies, and data inputs. With usage-based pricing, finance teams find it difficult to predict costs. And even when predictions can be made, finance teams struggle to understand the value that is being obtained from the use.

Spend is also fragmented in ways that make it genuinely difficult to see. McKinsey found that 20-30% of AI spend is routinely unaccounted for, scattered across cloud providers, model vendors, software platforms, experimentation environments, and individual business units with no consolidated view. In one McKinsey engagement, a budget exercise uncovered costs spread across enterprise copilots, foundation-model contracts, AI-enabled software features, API-based services, and business-unit purchases. None of it was reconciled against a single source of truth. Some organizations have started taking down their internal AI usage leaderboards. The "go, go, go" energy of the previous two years is running into finance's first real look at the invoices.

The routing gap: premium pricing for commodity tasks

Without an intelligent routing layer, every request your agents make goes to the same frontier model, whether that request involves complex multi-step reasoning or is a simple data lookup that a faster, cheaper model handles just as well. McKinsey is direct on this: few tasks actually warrant frontier model pricing, and the biggest savings usually come from optimizing the systems consuming tokens, not from negotiating model rates.

The routing research makes this concrete. RouteLLM (ICLR 2025) found that well-trained complexity routers achieve 95% of frontier model performance while sending only 14-26% of requests to the expensive model, a 75-85% cost reduction on routed workloads. Most organizations aren't close to that because they haven't built the routing layer that would make it possible. Everything lands at the top of the model hierarchy by default, and that default is expensive.

Agentic workflows make this worse faster than most teams expect. A workflow that generates ten model calls per interaction (where your budget assumption was one), multiplies the routing problem by 10x. For any organization running agents in production, model selection is the single highest-leverage optimization available. Most teams have no mechanism for making that selection dynamically, which means they're paying frontier prices across the board.

The data access gap: context bloat and schema over-exposure

The data access gap is less visible than routing, but for data-heavy workflows it often drives more waste. When your AI agents fetch enterprise data without a schema-aware access layer, they get everything: full cross-system schemas, unfiltered record sets, surrounding metadata. The model processes all of it, even when the actual request needed a fraction. Every unnecessary token in that context window is money spent on data your agent never required.

McKinsey names the fix directly: "Shorten prompts, limit context windows, pass only relevant sections of documents or tool outputs... Reusing static prompt context can reduce repeated input-token costs by up to about 90 percent, especially for retrieval-augmented generation and agents with large, stable prefixes." A governed data access layer is the architectural implementation of that principle: schema awareness and scoped data return enforced at the infrastructure level, rather than left to individual developers to hand-tune for every request.

Tool design adds a second cost reduction on top of this. When tool calls are filtered to the requesting user's authorization or handle request complexities like aggregations or cross-system joins before the request reaches the model, the model only receives the records that user is allowed to see, limited to the minimum needed to answer the question. That eliminates both the token cost of processing out-of-scope data and the risk of the model surfacing records outside the user's permissions. That's the same architectural constraint solving two problems at once. CData's own benchmarks found a 97.6% reduction in token consumption on governed multi-source enterprise queries compared to unfiltered access. That figure reflects the same pattern McKinsey describes: send only what the request needs, filtered to what the user is authorized to receive.

The governance gap: invisible spend and no accountability

Only 20-25% of companies have mature AI FinOps practices in place, according to McKinsey. For most organizations, cost overruns become visible only after the invoice arrives, which is by definition after the optimization window has closed.

The accountability problem has a concrete operational shape. Which team generated that overage last month? Which agent workflow looped 500 times unnecessarily? Which fetch returned 50,000 rows when your use case needed 12? Without cost attribution at the request level, tagged to team, use case, and workflow, there's no way to trace waste back to its source, let alone prevent it from recurring. Attribution infrastructure is the prerequisite for optimization work, not an optional reporting layer to add once things are more settled.

McKinsey recommends building governance into the architecture via AI gateways, control planes, policy engines, and automated guardrails, rather than relying on behavioral change from individual teams. They also recommend shifting toward cost-per-outcome metrics: cost per claim processed, cost per resolved request, cost per customer interaction. That kind of accountability requires infrastructure that connects AI consumption to specific business activities, not just aggregate API invoices. The measurement model and the governance layer aren't two separate requirements. They're the same one.

What cost-managed production AI actually looks like

About a third of organizations surveyed by McKinsey have already achieved 20-30% cost reductions through active optimization. What separates them from the majority isn't discipline. It's architecture. Governance is built into the system, not added as a reporting layer after costs have already run.

Cost-managed AI deployments share four structural components: intelligent routing that directs each request to the appropriate model based on complexity and cost; schema-aware data access and tools that return only the relevant, permission-filtered data the request requires; cost attribution that tags every token to a team, workflow, and use case; and budget enforcement that surfaces runaway spend before it shows up on next month's invoice. Organizations getting meaningful cost reductions treat these as infrastructure, not as four sequential projects to work through when bandwidth allows.

That's where CData Connect AI comes in. Connect AI provides schema-aware, permission-filtered access to hundreds of enterprise data sources through a single MCP-compliant interface, cutting context bloat on data-heavy queries and generating the per-request audit logs that enable cost attribution at the data layer. The cost efficiency and the accountability infrastructure come from the same architectural layer, which means your teams don't have to choose between the two.

Building the business case for AI governance infrastructure

McKinsey puts the numbers plainly: organizations with high forecasting maturity save 10% more on AI spend than peers, and those that actively optimize capture 20-30% reductions. For a company spending $5 million a year on AI, that's $1 million - $1.5 million in recoverable spend. The relevant question isn't whether governance infrastructure costs money. It does. It's whether that cost is smaller than the waste it prevents.

Most engineering teams considering this face a build-versus-buy question. Building cost attribution, routing logic, and schema-aware data access in-house requires real engineering time and ongoing maintenance as model APIs change, schemas evolve, and agent architectures shift. That burden grows with every additional data source, model provider, and production workflow. Teams that have tried it tend to end up with something brittle: a bespoke system tuned to the current model that can't adapt when a cheaper alternative emerges six months later.

McKinsey recommends standing up a permanent AI FinOps function: a cross-functional team that forecasts demand, monitors consumption, finds optimization opportunities, and tracks cost per outcome. Good governance infrastructure is what gives that team something to work with. It's also what keeps your organization from being locked into a particular model or vendor. The model landscape has shifted significantly in the past 18 months alone. Organizations with governance infrastructure in place can move workloads when a better or cheaper option appears. Those without it tend to be stuck.

Frequently asked questions

Reduce enterprise AI token costs at the data access layer

CData Connect AI is the enterprise AI data gateway that eliminates context bloat at the source, providing schema-aware, permission-filtered access to hundreds of enterprise data sources through a single MCP-compliant interface. Every request returns only what the model needs, with full audit logging for cost attribution.