Microsoft Fabric Machine Learning Tutorial - Part 5 - Observability

In part 5 of this course Barry Smart, Director of Data and AI, walks through a demo showing how to improve observability of your data engineering processes using readily available technology and platforms in with Microsoft Fabric and Azure.

Barry begins the video by explaining that one of the pitfalls of using Fabric Notebooks in an operational setting is that it can impede observability of what is going on within the notebook. To address this he shows how standard Python Logging, combined with Azure Application Insights can be used to emit telemetry from key points in your notebook and captured centrally. We demonstrate this approach with a create_logger function, showcasing its application in a Fabric notebook and the benefits of centralizing logs.

This episode helps ensure continuous monitoring and rapid issue detection for optimal data product performance. Application Insights then provides to the tools ot enable you to proactively monitor the behaviour of your entire data estate to sustain value over the long term. It also provides a valuable source of troubleshooting information when things do go wrong to help you quickly diagnose and resolve issues.

Stay tuned for our next episode on deploying resources through infrastructure as code.

Chapters:

• 00:00 Series Recap
• 00:34 Focus on Observability
• 02:33 Implementing Logging in Fabric
• 04:01 Demo: Logging in Action
• 05:26 Monitoring with Application Insights
• 06:02 Conclusion and Next Steps
• 08:45 Final Thoughts and Next Steps

From Descriptive to Predictive Analytics with Microsoft Fabric:

• Part 1 - Overview
• Part 2 - Data Validation with Great Expectations
• Part 3 - Testing Notebooks
• Part 4 - Task Flows
• Part 5 - Observability

Microsoft Fabric End to End Demo Series:

• Part 1 - Lakehouse & Medallion Architecture
• Part 2 - Plan and Architect a Data Project
• Part 3 - Ingest Data
• Part 4 - Creating a shortcut to ADLS Gen2 in Fabric
• Part 5 - Local OneLake Tools
• Part 6 - Role of the Silver Layer in the Medallion Architecture
• Part 7 - Processing Bronze to Silver using Fabric Notebooks
• Part 8 - Good Notebook Development Practices

Microsoft Fabric First Impressions:

• Perspectives on Microsoft Fabric
• Introduction to Microsoft Fabric
• A Tour Around Microsoft Fabric

Decision Maker's Guide to Microsoft Fabric

• Hedging your Fabric Bet
• Microsoft Fabric and Generative AI
• The Pace of Innovation

and find all the rest of our content here.