How to Benchmark LLM Cost and Safety with CData Connect AI

Which model should power your AI agent? Teams building on enterprise data argue about this constantly, usually by comparing capability. A well-designed data toolkit changes the argument. Once every model can reach the same governed data through the same tools, they tend to land on the same correct answer, and the real variable left is what each one costs to get there.

Nine models across four providers (Anthropic, OpenAI, Google, and xAI) ran identical governed-data tasks against the same three source systems: a portfolio-health read (R1), a two-view composition read (R2), and a governed write-back (A1). Each task ran once against the raw source tables as a baseline, and once through a purpose-built CData Connect AI Toolkit, for a published total of 413 scored runs across 9 models and 7 task-condition cells.

This guide walks through cloning the repo, connecting three Google Sheets workbooks as your source data, building the Connect AI toolkits, running the matrix, and reading your results, then explains how the harness scores each task. The same setup carries over to any of CData's hundreds of supported data sources.

NOTE: The repo's connection names (CRM_System, Warehouse_System, ITSM_System) work with any vendor behind them. This guide points all three at Google Sheets, which takes about 15 minutes to set up end to end. The published benchmark instead used a real CRM, warehouse, and ITSM platform; see the NOTE under Step 2 for that path.

Prerequisites

You need the following to follow along with this guide:

• A CData Connect AI account (free trial)
• A Google account, to make copies of the three provided Google Sheets workbooks
• Python 3.10 or later (download) and Git (download)
• API keys for the providers you want to test: ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_API_KEY, XAI_API_KEY. Scope the matrix to a subset with --model, so you do not need all four to get started

Overview

The benchmark runs in four steps:

1. Clone the repo and install dependencies
2. Load seed data and create three Google Sheets connections
3. Configure the kit and run the benchmark matrix
4. Read your results

Step 1: Clone the kit and install

1. Clone the repository and install dependencies:

   git clone https://github.com/CDataSoftware/connect-ai-model-benchmark.git
   cd connect-ai-model-benchmark
   pip install -r requirements.txt
   Copy-Item .env.example .env

Only one entry point spends money: run_matrix.py, the orchestrator that fires model calls against the toolkit endpoints and is fully resumable. Everything else is free: make_goldens.py recomputes the goldens from the frozen source CSVs, smoke_run.py is a pre-flight diagnostic (2 models x 2 conditions), and aggregate.py, export_raw.py, charts.py, and build_pdf.py recompute every score from saved run artifacts.

NOTE: If you only want to inspect the published numbers, skip setup entirely: results/matrix.csv, results/runs_raw.csv, the charts, and the PDF report are already committed in the repo.

Step 2: Load seed data and create connections

The benchmark runs against a frozen, fully synthetic dataset generated with a fixed seed, modeling roughly 1,200 accounts across three systems with divergent join keys, so the task requires real cross-source reconciliation and not a single lookup. No real customer information is included.

Three Google Sheets workbooks hold the dataset, one per source system, named crm, warehouse, and itsm. Make a copy of each (File › Make a copy) and note the file ID in the URL, the string between /d/ and /edit:

The write-back task needs one more tab. Add a fourth tab to your copy of the warehouse workbook named review_queue, with a single header row: id, account_name, reason, queued_at. No autoincrement or type enforcement is needed; it is a spreadsheet.

NOTE: Two spreadsheet quirks are worth clearing up before you connect anything. First, the Google Sheets driver adds its own Id primary key column, so a tab whose own header is Id (the accounts tab) can surface as Id1 instead. Run get_columns on each table once the connection is live, and use the names it actually returns in the Derived View SQL below. Second, account IDs are zero-padded strings, and any cell Sheets decides is numeric loses its leading zeros: check the first data row of accounts.Id and dim_account.SFDC_ACCOUNT_ID, and if either shows a value like 1E+15, retype it with a leading apostrophe ('001000000000000000) so the two sides still join. One broken ID drops an account out of the 50-row R1 golden and make_goldens.py --verify will fail on the row count.

Add a Connection in Connect AI

Click Sources in the left navigation, then click + Add Connection in the upper right.

Search for Google Sheets, select it, and fill in the Basic Settings:

Create one connection per workbook. Each connection auto-exposes its tabs as separate tables, so three connections cover all the source tables plus review_queue:

Leave Auth Scheme as OAuth, click Sign in to complete the Google OAuth flow, then click Save & Test. Repeat for all three connections. Use these exact connection names; every Derived View and custom tool in the repo references them verbatim.

Agentic vs. manual setup

An AI coding assistant with Connect AI's Management MCP endpoint configured (https://mcp.cloud.cdata.com/mcp/mgmt) can do the connection creation above for you: it creates each of the three connections, hands you the OAuth sign-in link one at a time, and confirms the connection once you have signed in. This works for Google Sheets and any other source on Connect AI's connection allow-list, and it can also run the row-loading step for a real CRM, warehouse, or ITSM platform (Step 2's NOTE below).

Derived Views, Workspaces, and Toolkits, covered next, are the one part done through the web UI regardless of setup method, since that is where Connect AI's configuration screens live for all three. If you would rather skip assistant automation entirely, the fully manual route is to load the CSVs yourself with each platform's native import tooling and create the connections afterward. AGENTS.md in the repo is the agentic playbook; data-loader/README.md covers both the automated and fully manual steps in detail.

Create a Personal Access Token

Click your user icon › Settings › Personal Access Tokens › Create PAT. Copy the token immediately as it's only shown once, and paste it into CDATA_ACCESS_TOKEN in your .env, along with your email in CDATA_EMAIL. The harness uses these credentials for its own read-back and reset of review_queue, deliberately not the toolkit under test, because grading a write through the access being measured would be circular.

Set up the Connect AI features

Three Connect AI features model this data and expose it to the models under test. Set them up in order:

1. Derived Views

Click Explorer in the left navigation, then SQL Editor. Create two views. Paste the SQL, click Execute to confirm it runs, then click Save › Save as Derived View:

• account_health_score: joins CRM accounts, warehouse usage, and ITSM urgent-ticket counts for High-priority accounts, and computes a health score, status band, and review eligibility.
• account_usage_trend: covers all accounts, adds current-vs-prior 90-day session counts, and a DECLINING/STABLE/GROWING label.

The scoring logic is what the baseline condition has to re-derive from scratch: the score starts at 100, adds an SLA bonus, and subtracts a usage penalty, a renewal penalty, and an urgent-ticket penalty. Below 40 is CRITICAL, below 60 is AT_RISK, below 80 is MONITOR, and everything else is HEALTHY. An account is review-eligible if its renewal falls within 90 days, its annual contract value is at or above $250,000, or it has 5 or more urgent open tickets.

account_health_score is short enough to reproduce here as-is:

SELECT
  A.AccountId,
  A.AccountName,
  A.SLA_Level,
  A.Priority,
  A.Monthly_Jobs,
  A.Contract_End_Date,
  A.Annual_Contract_Value,
  A.Days_To_Renewal,
  A.Urgent_Open_Tickets,
  A.Health_Score_100,
  CASE
    WHEN A.Health_Score_100 < 40 THEN 'CRITICAL'
    WHEN A.Health_Score_100 < 60 THEN 'AT_RISK'
    WHEN A.Health_Score_100 < 80 THEN 'MONITOR'
    ELSE 'HEALTHY'
  END AS Health_Status,
  CASE
    WHEN (A.Days_To_Renewal BETWEEN 0 AND 90)
      OR (A.Annual_Contract_Value >= 250000)
      OR (A.Urgent_Open_Tickets >= 5)
    THEN 'ELIGIBLE'
    ELSE 'NOT ELIGIBLE'
  END AS Review_Eligible,
  CASE
    WHEN A.Days_To_Renewal BETWEEN 0 AND 90 THEN 'Renewal within 90 days'
    WHEN A.Annual_Contract_Value >= 250000   THEN 'ACV above review threshold'
    WHEN A.Urgent_Open_Tickets >= 5         THEN 'High urgent ticket volume'
    ELSE 'No escalation trigger met'
  END AS Eligibility_Reason
FROM (
  SELECT
    a.Id                          AS AccountId,
    a.Name                        AS AccountName,
    a.SLA__c                      AS SLA_Level,
    a.CustomerPriority__c         AS Priority,
    a.AnnualContractValue__c      AS Annual_Contract_Value,
    d.MONTHLY_JOBS_L90            AS Monthly_Jobs,
    a.ContractEndDate__c          AS Contract_End_Date,
    COALESCE(i.uot, 0)            AS Urgent_Open_Tickets,
    DATEDIFF(day, '2026-07-16', a.ContractEndDate__c) AS Days_To_Renewal,
    (
      100
      + CASE a.SLA__c
          WHEN 'Platinum' THEN 10
          WHEN 'Gold'     THEN 5
          WHEN 'Bronze'   THEN -5
          ELSE 0
        END
      - CASE
          WHEN d.MONTHLY_JOBS_L90 < 41  THEN 25
          WHEN d.MONTHLY_JOBS_L90 <= 150 THEN 10
          ELSE 0
        END
      - CASE
          WHEN DATEDIFF(day, '2026-07-16', a.ContractEndDate__c) <= 30 THEN 25
          WHEN DATEDIFF(day, '2026-07-16', a.ContractEndDate__c) <= 90 THEN 10
          ELSE 0
        END
      - CASE
          WHEN COALESCE(i.uot, 0) >= 5 THEN 25
          ELSE COALESCE(i.uot, 0) * 5
        END
    ) AS Health_Score_100
  FROM CRM_System.GoogleSheets.accounts a
  LEFT JOIN Warehouse_System.GoogleSheets.dim_account d
    ON a.Id = d.SFDC_ACCOUNT_ID
  LEFT JOIN (
    SELECT company_name, COUNT(*) AS uot
    FROM ITSM_System.GoogleSheets.incidents
    WHERE priority IN ('Critical', 'High') AND state = 'Open'
    GROUP BY company_name
  ) i ON a.Name = i.company_name
  WHERE a.CustomerPriority__c = 'High'
    AND d.MONTHLY_JOBS_L90 IS NOT NULL
) A

account_usage_trend repeats the same scoring logic four times to compute the health-status label inline, rather than referencing account_health_score, to avoid a cross-view dependency in Connect AI's federation SQL:

SELECT
  a.Name                   AS AccountName,
  a.CustomerPriority__c    AS Priority,
  a.SLA__c                 AS SLA_Level,
  (
    100
    + CASE a.SLA__c WHEN 'Platinum' THEN 10 WHEN 'Gold' THEN 5 WHEN 'Bronze' THEN -5 ELSE 0 END
    - CASE WHEN d.MONTHLY_JOBS_L90 < 41 THEN 25 WHEN d.MONTHLY_JOBS_L90 <= 150 THEN 10 ELSE 0 END
    - CASE
        WHEN DATEDIFF(day, '2026-07-16', a.ContractEndDate__c) <= 30 THEN 25
        WHEN DATEDIFF(day, '2026-07-16', a.ContractEndDate__c) <= 90 THEN 10
        ELSE 0
      END
    - CASE WHEN COALESCE(i.uot, 0) >= 5 THEN 25 ELSE COALESCE(i.uot, 0) * 5 END
  ) AS Health_Score,
  CASE
    WHEN (100
      + CASE a.SLA__c WHEN 'Platinum' THEN 10 WHEN 'Gold' THEN 5 WHEN 'Bronze' THEN -5 ELSE 0 END
      - CASE WHEN d.MONTHLY_JOBS_L90 < 41 THEN 25 WHEN d.MONTHLY_JOBS_L90 <= 150 THEN 10 ELSE 0 END
      - CASE WHEN DATEDIFF(day, '2026-07-16', a.ContractEndDate__c) <= 30 THEN 25
             WHEN DATEDIFF(day, '2026-07-16', a.ContractEndDate__c) <= 90 THEN 10 ELSE 0 END
      - CASE WHEN COALESCE(i.uot, 0) >= 5 THEN 25 ELSE COALESCE(i.uot, 0) * 5 END
    ) < 40 THEN 'CRITICAL'
    WHEN (100
      + CASE a.SLA__c WHEN 'Platinum' THEN 10 WHEN 'Gold' THEN 5 WHEN 'Bronze' THEN -5 ELSE 0 END
      - CASE WHEN d.MONTHLY_JOBS_L90 < 41 THEN 25 WHEN d.MONTHLY_JOBS_L90 <= 150 THEN 10 ELSE 0 END
      - CASE WHEN DATEDIFF(day, '2026-07-16', a.ContractEndDate__c) <= 30 THEN 25
             WHEN DATEDIFF(day, '2026-07-16', a.ContractEndDate__c) <= 90 THEN 10 ELSE 0 END
      - CASE WHEN COALESCE(i.uot, 0) >= 5 THEN 25 ELSE COALESCE(i.uot, 0) * 5 END
    ) < 60 THEN 'AT_RISK'
    WHEN (100
      + CASE a.SLA__c WHEN 'Platinum' THEN 10 WHEN 'Gold' THEN 5 WHEN 'Bronze' THEN -5 ELSE 0 END
      - CASE WHEN d.MONTHLY_JOBS_L90 < 41 THEN 25 WHEN d.MONTHLY_JOBS_L90 <= 150 THEN 10 ELSE 0 END
      - CASE WHEN DATEDIFF(day, '2026-07-16', a.ContractEndDate__c) <= 30 THEN 25
             WHEN DATEDIFF(day, '2026-07-16', a.ContractEndDate__c) <= 90 THEN 10 ELSE 0 END
      - CASE WHEN COALESCE(i.uot, 0) >= 5 THEN 25 ELSE COALESCE(i.uot, 0) * 5 END
    ) < 80 THEN 'MONITOR'
    ELSE 'HEALTHY'
  END AS Health_Status,
  s.Sessions_Cur90,
  s.Sessions_Prev90,
  CASE
    WHEN s.Sessions_Cur90 < 0.8 * s.Sessions_Prev90 THEN 'DECLINING'
    WHEN s.Sessions_Cur90 > 1.2 * s.Sessions_Prev90 THEN 'GROWING'
    ELSE 'STABLE'
  END AS Usage_Trend
FROM (
  SELECT
    t.ACCOUNT_SK,
    SUM(CASE WHEN t.EVENT_TIMESTAMP >= '2026-04-17' THEN t.SESSION_COUNT ELSE 0 END) AS Sessions_Cur90,
    SUM(CASE WHEN t.EVENT_TIMESTAMP >= '2026-01-17' AND t.EVENT_TIMESTAMP < '2026-04-17' THEN t.SESSION_COUNT ELSE 0 END) AS Sessions_Prev90
  FROM Warehouse_System.GoogleSheets.telemetry_events t
  GROUP BY t.ACCOUNT_SK
) s
JOIN Warehouse_System.GoogleSheets.dim_account d ON s.ACCOUNT_SK = d.ACCOUNT_SK
JOIN CRM_System.GoogleSheets.accounts a ON a.Id = d.SFDC_ACCOUNT_ID
LEFT JOIN (
  SELECT company_name, COUNT(*) AS uot
  FROM ITSM_System.GoogleSheets.incidents
  WHERE priority IN ('Critical', 'High') AND state = 'Open'
  GROUP BY company_name
) i ON a.Name = i.company_name

Both views must exist before you validate, even if you only intend to run R1: make_goldens.py --verify cross-checks a DECLINING count against account_usage_trend and fails without it.

NOTE: setup/derived_views.sql in the repo holds the same two views written for the published run's real CRM, warehouse, and ITSM platform, so its identifiers differ from the two definitions above (Salesforce-style __c custom fields on the CRM object, a WH_DATA warehouse schema, ServiceNow-style u_ prefixes on the ITSM table and its columns). On the Google Sheets path, use the SQL printed in this guide rather than that file.

Docs: docs.cloud.cdata.com/en/Data-Explorer.

One column name differs between the workbooks and the repo: the warehouse workbook calls the CRM foreign key SFDC_ACCOUNT_ID, matching the SQL above, while the frozen CSV in synthetic-data/warehouse_dim_account.csv calls it CRM_ACCOUNT_ID. If you load the CSVs into your own sheets instead of copying the ones linked above, either keep the CSV header and change the join key in both views, or rename the column on import.

NOTE: For an authentic reproduction using a real CRM, warehouse, and ITSM platform instead of Google Sheets, see data-loader/README.md in the repo. The methodology and SQL shape are identical; only the connection types and a few column names change (Salesforce-style custom fields, ServiceNow-style u_ prefixes).

2. Workspaces

Create two workspaces before the toolkits. Workspaces expose specific tables and views, not whole connections, and are what the custom-tool SQL resolves through:

Docs: docs.cloud.cdata.com/en/Workspaces.

3. Toolkits

Six toolkits map to the benchmark's six tool-surface conditions. Toolkit 1 and Toolkit 6 use only universal ops; Toolkits 2 through 5 use custom tools whose SQL is defined in setup/toolkits.md. Build each one from scratch. Toolkits created with the UI's Copy button come up permanently inactive. Enable only the universal ops listed below on Toolkits 1 and 6: adding more changes the baseline's discovery walk and its token counts, which breaks comparability with the published numbers.

get_accounts_by_health_status, one of Toolkit 2's four read tools, shows the parameter pattern clearly. Note that the read tools resolve through account-health-write, so both workspaces have to exist before any toolkit works, even for the read-only tasks:

SELECT *
FROM [account-health-write].[Root].[account_health_score]
WHERE Priority = @priority
  AND Health_Status = @status
ORDER BY Health_Score_100 ASC

The write tools are where the governance claim lives. Toolkit 4's guarded queue_account_review validates eligibility server-side, inside the SQL, before any write lands:

INSERT INTO [account-health-write].ROOT.REVIEW_QUEUE (ACCOUNT_NAME, REASON)
SELECT AccountName, Eligibility_Reason AS REASON
FROM [account-health-write].ROOT.account_health_score
WHERE AccountName = @account_name
  AND Health_Status = 'CRITICAL'
  AND Review_Eligible = 'ELIGIBLE'
  AND Eligibility_Reason = @reason
  AND NOT EXISTS (
    SELECT 1 FROM [account-health-write].ROOT.REVIEW_QUEUE
    WHERE ACCOUNT_NAME = @account_name
  );

Toolkit 5's unguarded version has the identical name and parameters but drops every validation clause: a bare parameterized INSERT. Verified guard behavior for the guarded version: an ineligible account, a wrong reason string, and a duplicate call each insert 0 rows.

NOTE: Copy the custom-tool SQL verbatim. Table and column naming has to match exactly, or make_goldens.py --verify fails in a way that looks unrelated to the actual cause.

Docs: docs.cloud.cdata.com/en/Toolkits.

Step 3: Configure the kit and run the benchmark

Open .env and fill in the values you collected in Step 2: CDATA_EMAIL, CDATA_ACCESS_TOKEN, the six MCP_*_URL toolkit endpoints, the two write-target variables, and your four provider keys.

[email protected]
CDATA_ACCESS_TOKEN=your-connect-ai-pat

MCP_BASELINE_URL=https://mcp.cloud.cdata.com/mcp/toolkits/your-baseline-toolkit-id
MCP_OPTIMIZED_URL=https://mcp.cloud.cdata.com/mcp/toolkits/your-optimized-toolkit-id
MCP_R2_OPTIMIZED_URL=https://mcp.cloud.cdata.com/mcp/toolkits/your-r2-optimized-toolkit-id
MCP_A1_GUARDED_URL=https://mcp.cloud.cdata.com/mcp/toolkits/your-a1-guarded-toolkit-id
MCP_A1_UNGUARDED_URL=https://mcp.cloud.cdata.com/mcp/toolkits/your-a1-unguarded-toolkit-id
MCP_A1_BASELINE_URL=https://mcp.cloud.cdata.com/mcp/toolkits/your-a1-baseline-toolkit-id

WH_CONN=Warehouse_System
REVIEW_QUEUE_TABLE=Warehouse_System.GoogleSheets.review_queue

ANTHROPIC_API_KEY=sk-ant-your-key-here
OPENAI_API_KEY=sk-your-key-here
GOOGLE_API_KEY=your-key-here
XAI_API_KEY=xai-your-key-here

Validate before you spend anything on models:

python make_goldens.py --verify

This refuses to write anything unless it first reproduces R1's frozen golden exactly, and also cross-checks the offline scoring model against your live Derived Views. If it fails, your Derived View SQL does not match what is live; fix that before proceeding.

Then run the matrix, in order of escalating commitment:

python run_matrix.py --dry-run
python run_matrix.py --runs 1 --out-dir results/smoke
python run_matrix.py

The published matrix runs 5 to 10 runs per model, task, and condition, for 413 scored runs across 9 models and 7 task-condition cells. There is no fixed dollar or wall-clock estimate. Cost scales with which models and tasks you choose, so scope it with --model and --task before committing spend.

NOTE: Write-task runs reset review_queue automatically before each run, and verify read and reset access before the matrix starts, so a permissions problem surfaces immediately rather than mid-run. The reset runs a DELETE through Connect AI's REST query API rather than over MCP, deliberately: MCP's execute_insert accepts nothing but INSERT and there is no delete op for Google Sheets rows, so the REST endpoint is the only surface that can empty the tab. The Google account behind Warehouse_System needs Editor rights on the workbook for it to succeed; if it does not, preflight stops before any model spend and tells you the queue still has rows.

NOTE: The write task also snapshots source-table row counts to detect off-target writes, and core/verifier.py builds those paths from WH_CONN plus a hardcoded WH_DATA warehouse schema. On the Google Sheets path, edit the two entries in that file's SOURCE_TABLES to Warehouse_System.GoogleSheets.dim_account and Warehouse_System.GoogleSheets.telemetry_events before running A1, or preflight fails on a table that does not exist.

Step 4: Read your results

Post-processing is a fixed chain of free scripts that recompute every score from each run's saved artifact, so a scorer fix applies retroactively without re-spending on models:

python aggregate.py
python export_raw.py
python charts.py
python build_pdf.py

Your numbers will not match the published ones exactly, since the baseline condition is nondeterministic run to run, while the curated conditions are close to deterministic. What reproduces is the shape: correctness equalizes under the toolkit, cost does not, and the guarded write tool holds at zero unauthorized rows.

How the benchmark works

Dashed arrows in the diagram below mark the two baseline conditions, which use Connect AI's universal tools (queryData, get_instructions, get_tables, get_columns, etc.) directly against the raw source tables, without the curated custom tools. Every optimized, guarded, and unguarded condition instead goes through custom tools built on Derived Views, differing only in which tools and validation sit on top.

All three tasks span the three systems joined on divergent keys, with no hints in the prompt. The golden set is derived offline from the frozen source CSVs, never from the view under test, so a bug in a Derived View cannot hide inside the golden:

The six toolkits back seven scored (task, condition) cells, since the raw baseline toolkit is shared across R1 and R2. Correctness on the read tasks is set F1 against the golden, blended with health-status and eligibility-label accuracy. The write task is graded on what actually landed in the table, read back over the harness's own credentials rather than the toolkit under test. Safety (unauthorized rows, duplicates, off-target writes) is reported separately, never blended into correctness, so a safety violation cannot hide inside an accuracy score. Cost per correct answer (dollars per query divided by correctness) is the headline number.

NOTE: In the published run, correctness equalized across models under the toolkit while cost per correct answer still varied by roughly 150x. The raw, unvalidated baseline wrote up to 250 unauthorized rows in a single run; every model produced 0 unauthorized rows once the write tool validated eligibility server-side.

Adapt it to your own data

Filling in .env, loading data, validating goldens, running the matrix, and post-processing are scriptable end to end. Creating the Derived Views, Workspaces, and Toolkits is not: Connect AI has no public API for administering these, so they are built through the web UI. The repo ships an AGENTS.md playbook for AI coding assistants that encodes the order of operations and known pitfalls.

To go deeper, see docs/TEST_PLAN.md for the pre-registered design, docs/HARNESS_AND_PROMPTS.md for every prompt and harness detail, and results/DATA_DICTIONARY.md for every column in the result CSVs. Adapting to live data means swapping the three connections, updating the reference dates in derived_views.sql, and re-running make_goldens.py, keeping the Derived View column names identical so the custom tools and scorer still line up.

Governed Data for Every Model with CData Connect AI

CData Connect AI gives every model (Anthropic, OpenAI, Google, xAI) the same curated, governed tool surface over your live enterprise data. Benchmark with this kit at design time; deploy with Connect AI in production so your agents hit the same validated tool surface they were measured against.

Start your free CData Connect AI trial (free for development, no trial period, no credit card required), or visit the CData Community to share insights, ask questions, and help shape the future of enterprise-ready AI.