Auditing UK energy policy without a cluster: a laptop, a duck, and twenty years of wind

Barry Smart, Director of Data and AI at endjin, sets out to audit UK energy policy with his dad, a fellow energy-industry veteran, on a single laptop. Their question: is the dash to Net Zero quietly compromising the security, reliability, and affordability of energy? They answer it not with opinion but with twenty years of fragmented government data.

Twenty-five years ago, work like this meant a fortune and a monolithic data warehouse. This time it is simple Python ingestion guarded by strict data contracts, a DuckDB medallion model, and requirements written in Gherkin as composable, fully tested functions, running where PySpark once failed to scale. The result: man-months of cost and "data fear" collapsed into a short laptop project, and a glimpse of how data teams freed from infrastructure can finally work as innovation teams.

Chapters

• 00:00 Introduction: endjin, DuckDB, and auditing UK energy policy
• 00:50 The data challenge and lessons from a 25-year-old data warehouse
• 01:40 Architecture: data contracts, Parquet, and a DuckDB medallion model
• 02:40 A data-driven approach with Gherkin and composable relations
• 03:40 Generating insights and why DuckDB scales where PySpark did not
• 04:30 New ways of working for data teams