Auditing UK energy policy without a cluster: a laptop, a duck, and twenty years of wind
Barry Smart, Director of Data and AI at endjin, sets out to audit UK energy policy with his dad, a fellow energy-industry veteran, on a single laptop. Their question: is the dash to Net Zero quietly compromising the security, reliability, and affordability of energy? They answer it not with opinion but with twenty years of fragmented government data.
Twenty-five years ago, work like this meant a fortune and a monolithic data warehouse. This time it is simple Python ingestion guarded by strict data contracts, a DuckDB medallion model, and requirements written in Gherkin as composable, fully tested functions, running where PySpark once failed to scale. The result: man-months of cost and "data fear" collapsed into a short laptop project, and a glimpse of how data teams freed from infrastructure can finally work as innovation teams.
Chapters
- 00:00 Introduction: endjin, DuckDB, and auditing UK energy policy
- 00:50 The data challenge and lessons from a 25-year-old data warehouse
- 01:40 Architecture: data contracts, Parquet, and a DuckDB medallion model
- 02:40 A data-driven approach with Gherkin and composable relations
- 03:40 Generating insights and why DuckDB scales where PySpark did not
- 04:30 New ways of working for data teams
Published on:
Learn moreWe can help you with Auditing UK energy policy without a cluster: a laptop, a duck, and twenty years of wind
If you want help implementing, troubleshooting, or improving this product, contact us and we’ll point you in the right direction.
Related posts
Async Rx.NET - The journey to GA
Ian Griffiths, Technical Fellow at endjin, shares the latest updates on the Async Reactive Extensions for .NET (AKA Async Rx.NET). Support for...
Introducing Survive the Sprint: a pixel-art game built (almost) in one shot with Claude Opus 5.5
I gave Claude Opus 5.5 one prompt and the endjin 'Who we are' page, and got back most of a pixel-art browser game starring the endjineers. Her...
How to trust your AI-assisted data analysis
AI tools can produce data analysis that looks authoritative without being verifiable. Whether you're the analyst or the decision-maker relying...
Providing comparative context with DAX Calculated Tables under Row Level Security
Row Level Security in Power BI solves one problem (who sees what) but introduces another - comparisons across the full dataset become impossib...
How to Implement Generation in RAG
Understand the generation step of RAG: how LLMs use augmented context to produce grounded responses, how to enforce structured outputs with Py...
How to Implement Augmentation in RAG
Understand the augmentation step of RAG: how retrieved documents are structured into prompts, how metadata and citations improve response qual...
How to Implement Retrieval in RAG
Understand the retrieval step of RAG: Learn how database queries, keyword search, vector search, and hybrid approaches find the right informat...
TypeDeclaration: An Abstraction for Understanding JSON Schema
The Corvus.Json.CodeGeneration library analyses JSON Schema and builds a TypeDeclaration tree that maps schema patterns to code patterns. The ...
Optimising DAX: Practical Examples
The final post in the Optimising DAX series: the CALCULATE trap, variables and IF.EAGER, slicer costs, and a practical approach to isolating s...
Optimising DAX: Data Materialisation
Data materialisation is when the storage engine gives up on efficient processing and rebuilds the entire table. This post explains what trigge...