How to ensure quality and avoid inaccuracies in your data insights

In this comprehensive session, discover how to ensure accuracy and reliability in your data projects by implementing testing methodologies for notebooks, pipelines, and Power BI reports.

James Broome, Director of Engineering, explores real-world examples from dozens of endjin's data & analytics projects; teaching techniques like unit testing in Fabric Notebooks, snapshot testing for ETL processes in Synapse Pipelines, and user acceptance testing for Power BI models. Learn to avoid common pitfalls and enhance your data practices with effective testing strategies.

Chapters

• 00:00 Introduction and Context
• 00:16 The Importance of Accurate Data Reporting
• 01:53 Challenges in Data Testing
• 03:40 Common Excuses for Lack of Testing
• 05:29 Levels of Software Testing
• 08:23 Unit Testing with Fabric Notebooks
• 11:28 Refactoring for Testability
• 19:09 Integration Testing with Pipelines
• 22:45 Testing Pipeline Behaviour with Valid and Invalid Data
• 23:40 Running and Monitoring Pipeline Tests
• 25:21 Snapshot Testing for Data Validation
• 31:05 User Acceptance Testing in Power BI
• 33:51 Creating Executable Specifications with Gherkin
• 34:40 Building Dynamic DAX Queries for Testing
• 37:18 Testing Power BI Reports Locally and in the Cloud
• 43:17 Conclusion and Final Thoughts