#data validation testing

[kiwiqa-softwaretesting]

Why Data Validation Testing Is Critical for AI and Analytics Platforms

Learn why data validation testing is vital for AI and analytics platforms to ensure accuracy, reliability & smarter business decisions acros

[etl-testing-tools]

Data Validation Testing

At a recent TDWI virtual summit on “Data Integration and Data Quality”, I attended a session titled “Continuous Data Validation: Five Best Practices” by Andrew Cardno.

In this session, Andrew Cardno, one of the adjunct faculty at TDWI talked about the importance of validating data from the whole to the part, which means that the metrics or total should be validated before reconciling the detailed data or drill-downs. For example, revenue totals by product type should be the same in Finance, CRM, and Reporting systems.

Attending this talk reminded me of a Data Warehouse project I worked on at one of the federal agencies. The source system was a Case Management system with a Data Warehouse for reporting. We noticed that one of the key metrics “Number of Cases by Case Type” yielded different results when queried on the source database, the data warehouse, and the reports. Such discrepancies undermine the trust in the reports and the underlying data. The reason for the mismatch can be an unwanted filter or wrong join or error during the ETL process.

When it comes to the federal agency this report is sent to congress and they have a congressional mandate to ensure that the numbers are correct. For other industries such as Healthcare and Financial, compliance requirements require the data to be consistent across multiple systems in the enterprise. It is essential to reconcile the metrics and the underlying data across various systems in the enterprise.

Andrew talks about two primary methods for performing Data Validation testing techniques to help instill trust in the data and analytics.

Glassbox Data Validation Testing

Blackbox Data Validation Testing

I will go over these Data Validation testing techniques in more detail below and explain how the Datagaps DataOps suite can help automate Data Validation testing.

[etl-testing-tools]

Datagaps DataOps suite is the most comprehensive data validation testing tool that can be used for Data Validating, Data Quality & Data Reconciliation testing.

I will go over these Data Validation testing techniques in more detail below and explain how the Datagaps DataOps suite can help automate Data Validation testing.

[etl-testing-tools]

This article discusses why and how to use both together, and dives into the challenges of Bulk Data Migration to Snowflake. Why and How? People are rapidly adopting cloud architectures for Data Warehouses and Machine Learning projects due to the economies of scale in the cloud. One obstacle in achieving rapid success is the data […]

People are rapidly adopting cloud architectures for Data Warehouses and Machine Learning projects due to the economies of scale in the cloud. One obstacle in achieving rapid success is the data and data types inconsistencies between on-premise structures and the modern data stacks in the cloud.

When you migrate vast amounts of data to the cloud, the opportunity to introduce mistakes is multiplied due to these reasons and others. The earlier you catch the issues, the less costly it is to resolve the discrepancies.