Data Engineering Reads

Starting with a local Parquet file, then joining it to data stored in the cloud The post Building a Data Lakehouse with DuckDB and DuckLake appeared first on To

Explore five free hands-on workshops covering data engineering, machine learning, MLOps, LLMs, AI agents, and AI development through practical lessons, homework

"Explain the Databricks architecture" is one of the most common data engineering questions. Here is...

Cloud data warehouse migrations are messy. You're pulling schemas out of legacy systems,...

Star schemas and SCDs get you started. Here's what actually breaks once a warehouse hits the real world — snowflake schemas, fact table types, junk dimensions,

Sixty organisations have joined an effort to make semantic models portable between tools. Snowflake started it. Databricks, dbt Labs… Continue reading on Toward

A hands-on guide to building an open lakehouse on Google Cloud: one Apache Iceberg table in GCS, three engines, real OCC conflict (HTTP 409) via the Lakehouse I

Deletion vectors, row lineage, a VARIANT type: Apache Iceberg v3 is GA on Snowflake and Databricks. Here is what each change means in production.

The AI wave has transformed the way how software professionals write code, allowing them to deliver fast with just a few prompts. However… Continue reading on M

Last October, a "minor" dependency bump in a PySpark job cost us four hours of downtime and roughly...

There was a time when building a data warehouse meant thinking about servers, storage capacity, database administration, infrastructure… Continue reading on Wri

How a single Python list controls the storage layout of ~180 CDC tables and why the busiest tables get the fewest buckets Continue reading on Medium »

The platforms have converged on paper. The right choice still comes down to your workloads, your team, and how each platform bills you for growth.

Initially, organizations relied heavily on data warehouses, storing structured data in traditional databases like MySQL and building… Continue reading on Medium

Today’s pipelines send alerts. Tomorrow’s pipelines will investigate, validate, and recover before engineers even wake up. Continue reading on Towards Data Engi

If you’ve spent any time around a data team, you’ve probably heard people throw around words like “data lake,” “warehouse,” “lakehouse,”… Continue reading on Me

Efficiently manage and terminate Spark sessions in PySpark using Apache Iceberg for optimized data workflows and resource utilization Continue reading on Medium

If you are working with Apache Airflow, Linux, automation, or data pipelines, you will eventually...

Every platform modernisation programme has the same first act, and it is never the interesting one. Before anyone gets to write a line of dbt , somebody has to

How I built multi-gate claim pipelines for long-horizon RL training tasks — taxonomy-driven scenario intake, AUTHOR / GOLDEN_DATA / REVIEWER roles, Auto-QC orch

Databricks has been moving data engineering toward a declarative model. Lakeflow Declarative Pipelines is the current framework for… Continue reading on Medium

Why PDFs, images and documents are becoming part of the modern lakehouse — and what this means for AI, RAG and data engineering Continue reading on Medium »

A data lake on S3-compatible storage is an object bucket holding open file formats (Parquet, ORC,...

A practical guide to building a data lakehouse on AWS — combining the flexibility of data lakes with the performance of data warehouses using Apache Iceberg, AW