When an AI assistant reports an incorrect sales pipeline total, finding the cause can be difficult. The assistant may have missed records because of limited access permissions, used outdated data, or relied on a mismatched schema. Each needs a different fix, so the first step is finding out how the assistant retrieved the data. An AI accuracy diagnostic checklist helps identify the source of the error and what needs to be corrected.
Before you start: how to reproduce and log wrong AI answers
MIT's NANDA initiative reported in "The GenAI Divide" that approximately 95% of organizations studied saw little or no measurable financial return from generative AI pilots. The report highlighted integration challenges, also discussed in CData's guide to choosing an enterprise AI gateway.
Start by collecting 10 to 20 wrong answers. For each one, log the exact prompt, the user identity, the timestamp, the AI tool, the answer it gave, and the correct value from the source system.
Tag each logged failure under one of three root cause categories, namely access and authorization, freshness and semantic context, or token and tool efficiency. The seven checks below are grouped the same way. Tagging failures before fixing them follows the Map and Measure functions of the National Institute of Standards and Technology (NIST) AI risk management framework, which helps present the findings within an established AI governance framework.
Data access and authorization checks (checks 1–2)
1. Confirm that user identity passes through to the source system (OAuth/SAML)
An AI assistant may return incomplete results or report that no records exist, even when the user can access them in the source application. This often happens when the assistant uses a shared service account with different permissions from the user, a pattern our guide to securing ChatGPT Enterprise connectors covers in another tool. Authorization is optional under the model context protocol (MCP) specification. When OAuth authorization is used, the MCP client acts as an open authorization (OAuth) 2.1 client to request access on behalf of the user. Check which identity and permissions the source system ultimately receives.
Microsoft states that Microsoft Copilot uses existing user permissions and only retrieves organizational data the user is authorized to view. To check whether permissions are causing an incorrect answer, run the same prompt using an affected user's account and an administrator's account. Differences in the results may indicate an access or identity issue.
CData Connect AI sends the end user's identity to the source over OAuth or security assertion markup language (SAML), which removes the service account problem.
2. Look for RBAC drift and delays in permission updates
The same prompt may return different answers across users or after a role change if permissions are not updated consistently across systems. For example, AWS Lake Formation checks access through both identity and access management (IAM) and Lake Formation permissions. To identify role-based access control (RBAC) inconsistencies, revoke a test user's permission and check how quickly the change takes effect in the AI assistant.
Encrypted and labeled content adds another layer. Where a sensitivity label applies encryption, Microsoft notes the user needs EXTRACT and VIEW usage rights for Copilot to summarize the content, so a user with view access alone can still get an incomplete answer. See our guide to enforcing source permissions at the connectivity layer for why a permissions model built on top of AI drifts from the source.
Data freshness and semantic context checks (checks 3–5)
3. Check whether the AI reads live data or outdated copies
An AI assistant may return figures that look correct but no longer match the source system. For example, closed deals may still appear open, or sales totals may reflect wrong data.
To check for outdated data, update a record in the source system and ask the assistant to retrieve it. If the change appears only after the next synchronization, the assistant is likely using a cached or replicated copy. Connect AI fetches data from the source on each request through its data access layer, with no copy in the path to fall behind.
4. Check for missing data models and flattened schemas that drop custom fields
Custom fields, such as a Salesforce Opportunity stage or region code, may be missing or incorrectly represented in AI responses. This can happen when unified APIs simplify source schemas and leave out custom objects or fields.
To identify missing fields, ask the assistant to list the fields available on an object and compare them with those in the source system. Connect AI discovers the full object model, including custom fields, rather than mapping to a fixed shape. See our guide to evaluating a context layer on schema intelligence and permission enforcement.
5. Find schema drift that breaks data extraction and AI inference
Schema drift occurs when fields in a source system are added, renamed, removed, or changed without downstream systems being updated. For example, a new custom field in Salesforce Opportunity may not appear in an AI tool that uses an older schema. Check when the tool last refreshed its schema, as cached definitions can cause it to miss fields or return incorrect results.
Token and tool efficiency checks (checks 6–7)
6. Count your MCP tools and test agents for tool overload
Custom AI agents may select the wrong tool, make unnecessary calls, or return inconsistent results. One cause is the number of tool definitions loaded into the model's context. Anthropic found that GitHub's 35 tools alone consumed approximately 26,000 tokens, with definitions across multiple servers potentially exceeding 50,000 tokens before processing a request. Using custom MCP tools to reduce token costs can help limit the number of tool definitions an agent needs to process.
To diagnose the issue, count the tools exposed across connected MCP servers and measure their token usage. Anthropic recommends considering deferred tool loading when there are 10 or more tools or definitions exceed roughly 10,000 tokens. CData also covers this approach in its guide to overcoming enterprise integration bottlenecks.
7. Measure how much of the context window goes to schema exploration
Measure how many tokens the system prompt, tool definitions, and schema exploration calls consume before the agent begins its task. Compare this with the model's total context window to identify unnecessary overhead. Intermediate tool results also consume tokens as data passes through the model, as explained in CData's guide to token efficient Claude workflows.
CData's internal benchmarks show how token usage and tool calls can vary across MCP implementations. In one test, conventional MCP servers used 96,000 tokens and 10 tool calls to complete a task, while Connect AI used 14,202 tokens and three calls.
In a separate CData Labs AI accuracy benchmark, Connect AI achieved 98.5% accuracy across 378 prompts, compared with 65% to 75% for other MCP approaches. We also tested 27 models across 810 runs, measuring tokens and tool calls per correct answer. The benchmark replication guide provides the resources to reproduce the tests.
How to explain your findings to IT and leadership
Lead with the finding that the failures trace to data access and context, and name which check each one failed. That tells leadership the problem is in the access path and which team owns the fix. Microsoft's documentation confirms that Copilot accesses organizational data based on the signed-in user's permissions. This helps identify whether the issue lies in the data access path and which team is responsible for fixing it.
Translate each failed check into business impact and name an owner, so the people affected can escalate without having to explain MCP.
Finding | Business impact | Owner |
RBAC drift | Reps see incomplete pipeline totals | IAM/identity team |
Stale replicated data | Forecasts built on yesterday's numbers | Data platform team |
Flattened schema | Custom fields missing from every answer | Integration owner |
Tool overload | Agents pick wrong tools, results vary | AI platform team |
Structure the report around the four functions of the NIST AI risk management framework, namely Govern, Map, Measure, and Manage. This gives leadership a clear process for assessing and addressing AI risks. Use the companion NIST AI 600-1 Generative AI profile to cover risks specific to generative AI. For each finding, record the failed check, supporting evidence, responsible team or owner, and target remediation date.
Fix AI accuracy at the source with CData Connect AI
CData Connect AI provides managed access to live enterprise data through the MCP. It supports user identity and source permissions through OAuth or SAML, retrieves data directly from connected systems, and discovers schemas, including custom objects and fields. These capabilities help address the access, freshness, and schema issues identified in checks 1, 3, 4, and 5. Centralized connectivity also reduces the need to manage separate tool sets for every source, helping limit the tool and token overhead covered in checks 6 and 7.
MCP requires servers to validate that access tokens were issued for them. Connect AI preserves user permissions without passing MCP tokens directly to source systems. Anthropic also recommends loading tools on demand to reduce token usage. In CData's internal benchmark, Connect AI used 14,202 tokens, compared with 96,000 tokens for conventional MCP servers.
To verify the improvements, rerun the 10 to 20 failed requests through Connect AI and compare the results with verified source data. You can start a free Connect AI trial to test with your own data.
Frequently asked questions
What should I check first when my AI gives wrong answers about company data?
Check the identity first. If the AI connects through a shared service account rather than the signed in user, its permissions no longer match the person asking, which explains most missing and partial answers.
Why does AI hallucinate numbers from systems like Salesforce or NetSuite even when it's connected?
A connection can still deliver the wrong data. One that reads a nightly copy, drops custom fields, or holds a schema captured at setup produces confident answers from incomplete data.
How do I know whether the problem is the AI model itself or the data it's being given?
Ask the same question through the AI and directly in the source system. If only the source is right, the gap is in data access. If both agree and the AI misreads it, the problem is reasoning.
What does "stale data" mean in an AI accuracy diagnostic, and why does it cause wrong answers?
Stale data is a cached or batch-synced copy standing in for live production data. The AI reasons correctly over numbers that were accurate at the last sync, so the answers look consistent and still disagree with the source.
How can I tell if too many tools or a missing schema context is causing my AI agent to return incorrect results?
Count the tools across every connected MCP server, measure the tokens used before the first request, then ask the agent to list the fields on one object. Wrong tool selection points at overload, missing fields at schema context.
What's the fastest way to run an AI accuracy diagnostic on an enterprise data request without disrupting production?
Use read only prompts against records you can verify, starting with the same question as a standard user and as an admin. Only check 2 modifies a permission, and that can run in a sandbox.
After I find the root cause, what should I do next to fix AI accuracy for company data long-term?
Fix the access path before tuning prompts or swapping models. Prompt changes cannot recover data the AI never reached.
