{"id":3890,"date":"2023-04-18T09:00:17","date_gmt":"2023-04-18T09:00:17","guid":{"rendered":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/?p=3890"},"modified":"2025-11-18T08:15:22","modified_gmt":"2025-11-18T08:15:22","slug":"accelerated-data-validation-with-wavicles-data-capture-tool","status":"publish","type":"post","link":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/case-studies\/accelerated-data-validation-with-wavicles-data-capture-tool\/","title":{"rendered":"Accelerated Data Validation With Wavicle\u2019s Data Capture Tool"},"content":{"rendered":"<p>Managing data is always difficult and leads to complex processes. Data quality and ETL testing are critical to validate, verify, and qualify data while preventing duplicate records and data loss. This process ensures data transfer from various heterogeneous environments to a central data repository while adhering to strict quality protocols and compliance regulations.<\/p>\n<p>However, data testing may require manual validation and auditing. This manual work increases the risk of defects leaking into the production environment without proper quality checks. Ultimately, it can be difficult for even the most talented and certified testers to create an accurate testing strategy to validate all possible data combinations and potential failures.<\/p>\n<h2>Project background<\/h2>\n<p>One of North America\u2019s largest collision repair center operators approached Wavicle for help building testing strategies and solutions within strict deadlines. They needed data validated across two stages: first, when moving from its source to a data lake, and second, when moving from a data lake to a data warehouse. The testing strategies included validating data in various file types and comparing it across the staging and warehouse layers.<\/p>\n<p>This project included several challenges:<\/p>\n<ul>\n<li><span style=\"font-size: 20px;\">An accelerated timeline of two months<\/span><\/li>\n<li><span style=\"font-size: 20px;\">Large data volumes with complex logic<\/span><\/li>\n<li><span style=\"font-size: 20px;\">Multiple file types, including XML, JSON, CSV &amp; Excel data comparison<\/span><\/li>\n<li><span style=\"font-size: 20px;\">Database-to-database validation for a process that included files coming from the business, then loading into a data lake (Athena), then moving into a data warehouse (RedShift) based on business logic, and then being translated onto dashboards<\/span><\/li>\n<\/ul>\n<h2>Wavicle\u2019s approach<\/h2>\n<p>The Wavicle team presented a data capture tool that compares various data sources and identifies variances in the same data group. Using multiple connectors to relational and non-relational databases, cloud databases, and filesystems, the tool compares the test data and generates test results into smart reports. The test results are then stored and accessed through user-defined Amazon S3 storage.<\/p>\n<p>This approach ensured the quality of the test deliverables and provided a better focus on testing. It paved a hassle-free path for testers to test high volumes of data containing more than one million records. With this system, testers could easily compare the results as the reports narrowed down the exact differences between the source data and target databases to identify bugs.<\/p>\n<p>Wavicle\u2019s team used their data capture accelerator to address a variety of challenges, including the following:<\/p>\n<ul>\n<li><span style=\"font-size: 20px;\"><strong>Complexity:<\/strong> The tool provides quality assessment to visualize test plans in workspaces to reduce the complexity of managing the test plans and retesting. Output-matching reports allowed for simpler evaluation of results and quicker understanding of actual errors for reporting.<\/span><\/li>\n<li><span style=\"font-size: 20px;\"><strong>Time:<\/strong> The tool significantly reduced the time needed to create test plans and corresponding SQL, verify the results, and communicate errors. This resulted in a 50% reduction in data validation efforts in complex data platforms. Proper testing of data flows can create hundreds of test steps for an average-size implementation, so time and cost savings can be significant.<\/span><\/li>\n<li><span style=\"font-size: 20px;\"><strong>Effort:<\/strong> Parameterizing the query for multiple batches in the same test plan reduced the effort that would be required to create multiple test plans.<\/span><\/li>\n<li><span style=\"font-size: 20px;\"><strong>Validation:<\/strong> Manually doing file-to-file validation took approximately 2 \u2013 3 hours. Execution with Wavicle\u2019s data capture accelerator reduced the time by 75% to validate within 30 minutes.<\/span><\/li>\n<li><span style=\"font-size: 20px;\"><strong>Defect identification:<\/strong> The tool rapidly identified bugs using a smart report.<\/span><\/li>\n<\/ul>\n<p>In addition, Wavicle\u2019s data capture tool helped by retesting defects. Using the data capture tool, no new test plans or additional efforts were needed after fixing defects, which reduced effort. Having the ability to edit the existing test plan in response to minor changes was faster and more straightforward than creating a new one. The test plans could also parameterize the queries, which enabled the team to run SQL queries for different conditions. This allowed them to run the test plans with different values for the conditions.<\/p>\n<p>During the project, Wavicle\u2019s team had to adapt to overcome unforeseen challenges. For example, they encountered issues with a server that went down while running high-volume test plans and deployed a fix to bring up the server automatically after 15 minutes of downtime and improve performance through coding solutions. In addition, handling XML and hierarchical JSON comparisons against CSV or DB was challenging. Using JSON conversion utilities, the team converted the XMLs to JSON, loaded them to MongoDB, and queried the MongoDB to flatten data, which enabled them to compare it against flat files like CSVs or DBs.<\/p>\n<h2>Project results<\/h2>\n<p>Wavicle\u2019s data capture tool reduced risk with the ability to manage hundreds of testing tasks. Plus, the tool made understanding test outputs and identifying key errors easier, allowing for faster and more reliable break\/fix cycles.<\/p>\n<p>Wavicle\u2019s consultants completed the two-month project on time, tested vast volumes of data in the order of a few hundred million records, and identified all data mismatches. This success was recognized and appreciated by the customer, as the rate of execution was three times faster than the manual execution, and the testing quality was noteworthy.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Migrated 550+ Tableau dashboards to QuickSight using EZConvertBI, achieving 80% automation, 60% time savings, and seamless collaboration under tight timelines and data constraints.<\/p>\n","protected":false},"author":2,"featured_media":5240,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[152,146,145,77,54,72,112,95],"tags":[],"class_list":["post-3890","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-amazon-athena","category-aws-redshift","category-amazon-s3","category-architecture-engineering","category-case-studies","category-data-management","category-mongo-db","category-technology","entry"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.0 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Accelerated Data Validation Using Wavicle\u2019s Capture Tool<\/title>\n<meta name=\"description\" content=\"Learn how Wavicle\u2019s data capture tool speeds up data validation processes, ensuring improved data quality and faster workflows.\" \/>\n<meta name=\"robots\" content=\"noindex, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Accelerated Data Validation Using Wavicle\u2019s Capture Tool\" \/>\n<meta property=\"og:description\" content=\"Learn how Wavicle\u2019s data capture tool speeds up data validation processes, ensuring improved data quality and faster workflows.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/case-studies\/accelerated-data-validation-with-wavicles-data-capture-tool\/\" \/>\n<meta property=\"og:site_name\" content=\"Wavicle Data Solutions\" \/>\n<meta property=\"article:published_time\" content=\"2023-04-18T09:00:17+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-11-18T08:15:22+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/wp-content\/uploads\/2023\/04\/13163.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1065\" \/>\n\t<meta property=\"og:image:height\" content=\"930\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Data Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Data Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/case-studies\/accelerated-data-validation-with-wavicles-data-capture-tool\/\",\"url\":\"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/case-studies\/accelerated-data-validation-with-wavicles-data-capture-tool\/\",\"name\":\"Accelerated Data Validation Using Wavicle\u2019s Capture Tool\",\"isPartOf\":{\"@id\":\"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/case-studies\/accelerated-data-validation-with-wavicles-data-capture-tool\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/case-studies\/accelerated-data-validation-with-wavicles-data-capture-tool\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/wp-content\/uploads\/2023\/04\/13163.jpg\",\"datePublished\":\"2023-04-18T09:00:17+00:00\",\"dateModified\":\"2025-11-18T08:15:22+00:00\",\"author\":{\"@id\":\"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/#\/schema\/person\/dd7976fcb051bcb40f39bb51c8fbdd9e\"},\"description\":\"Learn how Wavicle\u2019s data capture tool speeds up data validation processes, ensuring improved data quality and faster workflows.\",\"breadcrumb\":{\"@id\":\"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/case-studies\/accelerated-data-validation-with-wavicles-data-capture-tool\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/case-studies\/accelerated-data-validation-with-wavicles-data-capture-tool\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/case-studies\/accelerated-data-validation-with-wavicles-data-capture-tool\/#primaryimage\",\"url\":\"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/wp-content\/uploads\/2023\/04\/13163.jpg\",\"contentUrl\":\"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/wp-content\/uploads\/2023\/04\/13163.jpg\",\"width\":1065,\"height\":930},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/case-studies\/accelerated-data-validation-with-wavicles-data-capture-tool\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Accelerated Data Validation With Wavicle\u2019s Data Capture Tool\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/#website\",\"url\":\"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/\",\"name\":\"Wavicle Data Solutions\",\"description\":\"and analytics consulting solutions\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/#\/schema\/person\/dd7976fcb051bcb40f39bb51c8fbdd9e\",\"name\":\"Data Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/e23491a3af11f4a98f6410b9023b80c0675cb87552af0094392cf966c0311a96?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/e23491a3af11f4a98f6410b9023b80c0675cb87552af0094392cf966c0311a96?s=96&d=mm&r=g\",\"caption\":\"Data Team\"},\"url\":\"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/author\/data-team\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Accelerated Data Validation Using Wavicle\u2019s Capture Tool","description":"Learn how Wavicle\u2019s data capture tool speeds up data validation processes, ensuring improved data quality and faster workflows.","robots":{"index":"noindex","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"og_locale":"en_US","og_type":"article","og_title":"Accelerated Data Validation Using Wavicle\u2019s Capture Tool","og_description":"Learn how Wavicle\u2019s data capture tool speeds up data validation processes, ensuring improved data quality and faster workflows.","og_url":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/case-studies\/accelerated-data-validation-with-wavicles-data-capture-tool\/","og_site_name":"Wavicle Data Solutions","article_published_time":"2023-04-18T09:00:17+00:00","article_modified_time":"2025-11-18T08:15:22+00:00","og_image":[{"width":1065,"height":930,"url":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/wp-content\/uploads\/2023\/04\/13163.jpg","type":"image\/jpeg"}],"author":"Data Team","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Data Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/case-studies\/accelerated-data-validation-with-wavicles-data-capture-tool\/","url":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/case-studies\/accelerated-data-validation-with-wavicles-data-capture-tool\/","name":"Accelerated Data Validation Using Wavicle\u2019s Capture Tool","isPartOf":{"@id":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/#website"},"primaryImageOfPage":{"@id":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/case-studies\/accelerated-data-validation-with-wavicles-data-capture-tool\/#primaryimage"},"image":{"@id":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/case-studies\/accelerated-data-validation-with-wavicles-data-capture-tool\/#primaryimage"},"thumbnailUrl":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/wp-content\/uploads\/2023\/04\/13163.jpg","datePublished":"2023-04-18T09:00:17+00:00","dateModified":"2025-11-18T08:15:22+00:00","author":{"@id":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/#\/schema\/person\/dd7976fcb051bcb40f39bb51c8fbdd9e"},"description":"Learn how Wavicle\u2019s data capture tool speeds up data validation processes, ensuring improved data quality and faster workflows.","breadcrumb":{"@id":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/case-studies\/accelerated-data-validation-with-wavicles-data-capture-tool\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/case-studies\/accelerated-data-validation-with-wavicles-data-capture-tool\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/case-studies\/accelerated-data-validation-with-wavicles-data-capture-tool\/#primaryimage","url":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/wp-content\/uploads\/2023\/04\/13163.jpg","contentUrl":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/wp-content\/uploads\/2023\/04\/13163.jpg","width":1065,"height":930},{"@type":"BreadcrumbList","@id":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/case-studies\/accelerated-data-validation-with-wavicles-data-capture-tool\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/"},{"@type":"ListItem","position":2,"name":"Accelerated Data Validation With Wavicle\u2019s Data Capture Tool"}]},{"@type":"WebSite","@id":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/#website","url":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/","name":"Wavicle Data Solutions","description":"and analytics consulting solutions","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/#\/schema\/person\/dd7976fcb051bcb40f39bb51c8fbdd9e","name":"Data Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/e23491a3af11f4a98f6410b9023b80c0675cb87552af0094392cf966c0311a96?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e23491a3af11f4a98f6410b9023b80c0675cb87552af0094392cf966c0311a96?s=96&d=mm&r=g","caption":"Data Team"},"url":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/author\/data-team\/"}]}},"_links":{"self":[{"href":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/wp-json\/wp\/v2\/posts\/3890","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/wp-json\/wp\/v2\/comments?post=3890"}],"version-history":[{"count":1,"href":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/wp-json\/wp\/v2\/posts\/3890\/revisions"}],"predecessor-version":[{"id":3891,"href":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/wp-json\/wp\/v2\/posts\/3890\/revisions\/3891"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/wp-json\/wp\/v2\/media\/5240"}],"wp:attachment":[{"href":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/wp-json\/wp\/v2\/media?parent=3890"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/wp-json\/wp\/v2\/categories?post=3890"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/beta74.thewebsitepreview.com\/wavicle\/dev\/wp-json\/wp\/v2\/tags?post=3890"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}