CData Virtuality 4.8 Release Notes: Resource Governor, Timezone Support for Jobs, New connectors, and More

This release comes packed with new features and improvements and also bug fixes.

The biggest new feature is the Resource Governor to help manage the Server load more efficiently by setting limits on maximum query execution time, number of rows returned by a query, and number of concurrent queries per user. Another major improvement is that jobs now can be run according to account-level and job-specific time zones thanks to a new 'timezone' parameter for schedules - this comes in handy if your team is working from different parts of the world. This feature also includes automatic adjustments for Daylight Saving Time. We've also expanded our foreign functions (i.e. user-defined and database-specific functions) functionality with support of Snowflake v8 functions. As part of this task, we've moved user-defined and predefined source-specific functions to the same system tables where other functions are located, meaning that metadata, definitions, change history, etc. for UDFs and source-specific functions have been moved to new and existing function-specific system tables. For example, function definitions have been moved to the new system table SYSADMIN.FunctionDefinitions.

For Snowflake, we've also introduced a new upload mode for loading data via the Snowflake internal stage and improved performance of updates and deletes in the SYSADMIN.historyUpdate procedure.

We've also updated the Google Ads API to v17 which means that "ad", "campaign_aggregate_asset_view", "channel_aggregate_asset_view", "local_services_employee", "offline_conversion_upload_conversion_action_summary", "shopping_product", and "user_list_customer_type" table have been added, some columns were removed, and some new columns were added to the tables. For full lists of removed and added objects, please ask our support team.

As for bug fixes, here are the most important ones: we've resolved a bug affecting the ConnectorManager which accumulated resources without releasing them afterwards, resulting in resource leakage; another bug with the FTP connector where the "saveFile" procedure calls failed on some FTP servers; another bug preventing the data source properties from being masked in logs when using the "SYSADMIN.createOrReplaceDatasource" procedure; and last but not least, a bug affecting the OAuth 2 Authorization where user limits did not work for SSO users.

For the Studio, we've fixed a bug causing an incorrect error message to be shown when "maxRows" limit is reached while scrolling through results using Page Down button; and for the Exporter, we've added export of user limits.

As for the connectors, we've added eight new ones: Apache Kafka, KsqlDB, Zoho CRM, SAP SuccessFactors, ActiveDirectory, Shopee, Snapchat, and Alteryx. For the existing connectors, most notably, we've added two improvements to Amazon Selling Partner, by improving error handling for Amazon Orders child tables and updating columns of the FBAManageInventoryHealth report, and fixed five bugs: Orders by Last Update Report not working for Australia region, Salesandtraffic report creating duplicate records, the FBAFeePreview procedure failing, duplicates in Flat File VAT Invoice Data Report (VIDR), and reports failing with the error "CANCELLED. The exception was thrown by the procedure 'amazon_sp.internal_ReportCreateWaitDownload'". For Amazon Ads, we've extended sponsored display with new metrics and fixed two bugs: one causing the Consolidated_Report to fail with "Method Not Found" error and another resulting in Report_Brands_PurchasedProducts failing with 403 error. We've also updated Snopify to API version 2024-07 and created best practice templates for eBay.

Here are all issues in this release:

Server

• DVCORE-8546 (New Feature): Implement the backend support of Time Zone Awareness for jobs
• DVCORE-8482 (New Feature): Add a per-user setting for query execution time limit
• DVCORE-8481 (New Feature): Add a per-user setting to limit the number of rows returned in a query
• DVCORE-8480 (New Feature): Add a per-user limit of concurrent queries
• DVCORE-8459 (New Feature): Add a procedure for importing user-defined functions
• DVCORE-8347 (New Feature): Snowflake: introduce a new upload mode for loading data via Snowflake internal stage
• DVCORE-8507 (Improvement): Snowflake: expand the list of supported built-in data source-specific functions with the top 20 Snowflake functions not yet covered by CData Virtuality
• DVCORE-8498 (Improvement): Snowflake: improve performance of updates and deletes in "SYSADMIN.historyUpdate" procedure
• DVCORE-8552 (Improvement): Improve ODBC metadata cache cleaner performance
• DVCORE-8545 (Improvement): Improve permission update handling to avoid thread blocking
• DVCORE-8543 (Improvement): Integrate Active Directory CData JDBC driver
• DVCORE-8542 (Improvement): Integrate SAP Success Factor CData JDBC driver
• DVCORE-8541 (Improvement): Integrate Zoho CRM CData JDBC driver
• DVCORE-8534 (Improvement): Update the Spring Beans library
• DVCORE-8512 (Improvement): Remove deprecated Java options from startup scripts
• DVCORE-8496 (Improvement): Integrate Apache Kafka CData JDBC driver
• DVCORE-8454 (Improvement): Google Ads API: update to v17
• DVCORE-8442 (Improvement): Move user defined and predefined source-specific functions to the same system tables where other functions are located
• DVCORE-8578 (Bug Fix): An infinite loop in SslConduit is caused by the bug in the Undertow
• DVCORE-8574 (Bug Fix): User limit changes are not reflected in user history tables
• DVCORE-8573 (Bug Fix): FTP connector: "saveFile" procedure calls fail on some FTP servers
• DVCORE-8568 (Bug Fix): HTTP requests fail with "Unsupported Content-Coding: agzip" due to outdated libraries
• DVCORE-8563 (Bug Fix): Exasol: NULL value from DOUBLE column is represented as zero
• DVCORE-8539 (Bug Fix): Exasol: "SYSADMIN.historyUpdate" procedure fails with "BatchUpdateException-data exception - invalid character value for cast" error if replicated table contains NULL values
• DVCORE-8558 (Bug Fix): Data source properties are not masked in logs when using the "SYSADMIN.createOrReplaceDatasource" procedure
• DVCORE-8551 (Bug Fix): Creation of user defined functions fails for Analytical Storage
• DVCORE-8530 (Bug Fix): CData Virtuality REST API: server returns an unclear error message when parallel queries limit is hit by a user
• DVCORE-8522 (Bug Fix): CData Virtuality REST API: using pagination results in 500 Internal Server error
• DVCORE-8529 (Bug Fix): Materialization jobs return an incomplete error message when the parallel queries limit is hit for a user
• DVCORE-8523 (Bug Fix): "SYSADMIN.setPermissions" procedure causes ConcurrentModificationException when adding and removing permissions is done in parallel
• DVCORE-8521 (Bug Fix): OAuth 2 Authorization: user limits do not work for SSO users
• DVCORE-8518 (Bug Fix): OAuth2 Authorization: it is impossible to re-map OAuth role to another CData Virtuality role
• DVCORE-8513 (Bug Fix): OAuth2 Authorization: permission removal fails if CData Virtuality role is mapped to OAuth role
• DVCORE-8511 (Bug Fix): Wrong error messages is returned via ODBC and JDBC when a query gets interrupted due to user limits
• DVCORE-8504 (Bug Fix): MAINTENANCE mode: UNION queries are prohibited when MAINTENANCE mode is enabled
• DVCORE-8502 (Bug Fix): ksqlDB: data source creation fails if "streamTimeout" translator parameter's value is quoted
• DVCORE-8500 (Bug Fix): ConnectorManager accumulates resources without releasing them afterwards, resulting in resource leakage
• DVCORE-8490 (Bug Fix): Clustering: multi-server configuration does not work or causes errors when used with HAProxy
• DVCORE-8488 (Bug Fix): Column Level Data Lineage: disabled data lineage creates heavy PlanNode objects causing excessive memory consumption
• DVCORE-8467 (Bug Fix): Creation of modular connector data source with auto-deployment of dependencies fails if it has already been deployed but one of its dependencies got undeployed afterward
• DVCORE-8457 (Bug Fix): "requestStates" collection in "ConnectorManager" object gets too big
• DVCORE-8445 (Bug Fix): Default descriptions for internal functions and function parameters are stored to "Description" columns of system tables instead of "OriginalDescription"
• DVCORE-8415 (Bug Fix): Snowflake: stored procedures cannot be run on CData Virtuality Server
• DVCORE-8288 (Bug Fix): Common Table Expressions: SELECT INTO from multiple CTEs fails when one of them is recursive
• DVCORE-8013 (Bug Fix): Modular connector data source gets recreated by the "SYSADMIN.createOrReplaceDatasource" procedure even if the definition has not changed

Exporter

• DVCORE-8509 (Improvement): CData Virtuality Exporter: add export of user limits
• DVCORE-8517 (Bug Fix): CData Virtuality Exporter: user limits are not exported for SSO users

Studio

• DVCORE-8547 (Improvement): CData Virtuality Studio: implement Time Zone Awareness for jobs
• DVCORE-8519 (Bug Fix): CData Virtuality Studio: incorrect error message is shown when "maxRows" limit is reached while scrolling through results using Page Down button

Connectors

• SQL-857 (New Feature): Shopee: create connector
• SQL-740 (New Feature): Snapchat: create connector
• SQL-155 (New Feature): Alteryx: create connector
• SQL-952 (Improvement): Amazon SP: improve error handling for Amazon Orders child tables
• SQL-951 (Improvement): Amazon SP: update columns of the FBAManageInventoryHealth report
• SQL-929 (Improvement): Emarsys: update getemailcampaignslist and getemailcampaign reports to include 'root_campaign_id' and 'version_name'
• SQL-912 (Improvement): Google Ads API: ensure compatibility with v17
• SQL-824 (Improvement): Shopify: update to API version 2024-07
• SQL-688 (Improvement): eBay: create best practice templates
• SQL-641 (Improvement): Amazon Ads: extend sponsored display with new metrics
• SQL-945 (Bug Fix): Salesforce: pipes are not created
• SQL-938 (Bug Fix): Salesforce: base connector and translator are incorrect
• SQL-943 (Bug Fix): Keepa: job fail with "Not in GZIP format" error
• SQL-942 (Bug Fix): Microsoft Dynamics BC: connector installation fails due to duplicate procedure names
• SQL-937 (Bug Fix): Amazon SP: Orders by Last Update Report does not work for Australia region
• SQL-922 (Bug Fix): Amazon SP: Salesandtraffic report creates duplicate records
• SQL-932 (Bug Fix): Amazon Selling Partner: the FBAFeePreview procedure fails
• SQL-928 (Bug Fix): Amazon Selling Partner: duplicates in Flat File VAT Invoice Data Report (VIDR)
• SQL-925 (Bug Fix): Amazon Selling Partner: reports fail with the error "CANCELLED. The exception was thrown by the procedure 'amazon_sp.internal_ReportCreateWaitDownload'"
• SQL-936 (Bug Fix): Microsoft Dynamics: some procedures fail with a dynamic SQL error
• SQL-933 (Bug Fix): LinkedIn: OrganizationalEntityShareStatistics without the time interval parameter only fetches data for the last 10 days added a reportData field
• SQL-923 (Bug Fix): LinkedIn: AdCreatives report returns incorrect amount of data
• SQL-927 (Bug Fix): Zendesk: the target table gets empty when there are issues with API access
• SQL-880 (Bug Fix): Amazon ads: Consolidated_Report fails with "Method Not Found" error
• SQL-874 (Bug Fix): Amazon ads: Report_Brands_PurchasedProducts failed with 403 error
• SQL-440 (Bug Fix): Airtable: some of the tables are not being parsed