Microsoft Fabric - Processing Bronze to Silver using Fabric Notebooks
Notebooks in Fabric are a lot like notebook experiences in other tools you're probably already familiar with. They allow us to write code in a variety of languages and create a commentary alongside our code using interactive cells. Fabric notebooks also have built in integration with Lakehouses, and provide a built-in filesystem that can be used to store arbitrary files somehow associated with the notebook.
Notebooks are our go-to tool for wrangling Bronze data into Silver, which tends to follow standard processing patterns. In this video we'll do a whirlwind tour of notebooks in Microsoft Fabric, and take a look at how our "Process to Silver" notebook takes shape. The full transcript is available below.
The talk contains the following chapters:
- 00:00 Intro
- 00:10 Creating a blank notebook
- 01:04 Mounting a Lakehouse in a Fabric Notebook
- 02:05 Querying files in a Lakehouse from a Fabric Notebook
- 03:28 Query PySpark DataFrames directly with SQL
- 04:12 Built-in notebook filesystem
- 05:14 Walkthrough of "Process to Silver" notebook - reading data and transforming
- 08:18 Walkthrough of "Process to Silver" notebook - merging and writing data
- 10:03 Walkthrough of "Process to Silver" notebook - reading back written data
- 10:15 Roundup & Outro
Useful links:
- 📖 Microsoft Fabric Notebooks
- 📖 Connect Lakehouses and Fabric Notebooks
- 📖 Built-in Notebook filesystem
- 📖 Delta Lake Merge
Microsoft Fabric End to End Demo Series:
- Part 1 - Lakehouse & Medallion Architecture
- Part 2 - Plan and Architect a Data Project
- Part 3 - Ingest Data
- Part 4 - Creating a shortcut to ADLS Gen2 in Fabric
- Part 5 - Local OneLake Tools
- Part 6 - Role of the Silver Layer in the Medallion Architecture
- Part 7 - Processing Bronze to Silver using Fabric Notebooks
Microsoft Fabric First Impressions:
Decision Maker's Guide to Microsoft Fabric
and find all the rest of our content here.
Published on:
Learn moreRelated posts
How to Implement Generation in RAG
Understand the generation step of RAG: how LLMs use augmented context to produce grounded responses, how to enforce structured outputs with Py...
How to Implement Augmentation in RAG
Understand the augmentation step of RAG: how retrieved documents are structured into prompts, how metadata and citations improve response qual...
How to Implement Retrieval in RAG
Understand the retrieval step of RAG: Learn how database queries, keyword search, vector search, and hybrid approaches find the right informat...
TypeDeclaration: An Abstraction for Understanding JSON Schema
The Corvus.Json.CodeGeneration library analyses JSON Schema and builds a TypeDeclaration tree that maps schema patterns to code patterns. The ...
Optimising DAX: Practical Examples
The final post in the Optimising DAX series: the CALCULATE trap, variables and IF.EAGER, slicer costs, and a practical approach to isolating s...
Optimising DAX: Data Materialisation
Data materialisation is when the storage engine gives up on efficient processing and rebuilds the entire table. This post explains what trigge...
Building API Reference Documentation From Code, Part 2: Under the Hood
A deep dive into the cross-assembly linking, PDB-based source links, TFM scanning, enrichment merging, and search indexing that power our API ...
Building API Reference Documentation From Code, Part 1: The Pipeline
We generate about 8,800 API reference pages from 16 libraries across two engine versions, with source links, TFM availability badges, and hand...
Writing Effective Copilot Instructions for Complex Codebases
A flat copilot-instructions.md works for small projects. When your codebase hits 500k lines, 19 projects, and multiple code generators, you ne...
Optimising DAX: The Formula Engine and Storage Engine
When you execute a DAX query, two engines divide up the work. Understanding the split between the formula engine and the storage engine is key...