Data Engineering

Starting with a local Parquet file, then joining it to data stored in the cloud The post Building a Data Lakehouse with DuckDB and DuckLake appeared first on To

Explore five free hands-on workshops covering data engineering, machine learning, MLOps, LLMs, AI agents, and AI development through practical lessons, homework

"Explain the Databricks architecture" is one of the most common data engineering questions. Here is...

Cloud data warehouse migrations are messy. You're pulling schemas out of legacy systems,...

Star schemas and SCDs get you started. Here's what actually breaks once a warehouse hits the real world — snowflake schemas, fact table types, junk dimensions,

Sixty organisations have joined an effort to make semantic models portable between tools. Snowflake started it. Databricks, dbt Labs… Continue reading on Toward

A hands-on guide to building an open lakehouse on Google Cloud: one Apache Iceberg table in GCS, three engines, real OCC conflict (HTTP 409) via the Lakehouse I

Deletion vectors, row lineage, a VARIANT type: Apache Iceberg v3 is GA on Snowflake and Databricks. Here is what each change means in production.

The AI wave has transformed the way how software professionals write code, allowing them to deliver fast with just a few prompts. However… Continue reading on M

Last October, a "minor" dependency bump in a PySpark job cost us four hours of downtime and roughly...

There was a time when building a data warehouse meant thinking about servers, storage capacity, database administration, infrastructure… Continue reading on Wri

How a single Python list controls the storage layout of ~180 CDC tables and why the busiest tables get the fewest buckets Continue reading on Medium »

The platforms have converged on paper. The right choice still comes down to your workloads, your team, and how each platform bills you for growth.

Initially, organizations relied heavily on data warehouses, storing structured data in traditional databases like MySQL and building… Continue reading on Medium

Today’s pipelines send alerts. Tomorrow’s pipelines will investigate, validate, and recover before engineers even wake up. Continue reading on Towards Data Engi

If you’ve spent any time around a data team, you’ve probably heard people throw around words like “data lake,” “warehouse,” “lakehouse,”… Continue reading on Me

Efficiently manage and terminate Spark sessions in PySpark using Apache Iceberg for optimized data workflows and resource utilization Continue reading on Medium

If you are working with Apache Airflow, Linux, automation, or data pipelines, you will eventually...

Every platform modernisation programme has the same first act, and it is never the interesting one. Before anyone gets to write a line of dbt , somebody has to

How I built multi-gate claim pipelines for long-horizon RL training tasks — taxonomy-driven scenario intake, AUTHOR / GOLDEN_DATA / REVIEWER roles, Auto-QC orch

Databricks has been moving data engineering toward a declarative model. Lakeflow Declarative Pipelines is the current framework for… Continue reading on Medium

Why PDFs, images and documents are becoming part of the modern lakehouse — and what this means for AI, RAG and data engineering Continue reading on Medium »

A data lake on S3-compatible storage is an object bucket holding open file formats (Parquet, ORC,...

A practical guide to building a data lakehouse on AWS — combining the flexibility of data lakes with the performance of data warehouses using Apache Iceberg, AW

Enterprise AI is moving beyond models. The real competition is about data, context, governance, and task ownership.

Introduction Continue reading on Towards AI »

Databricks was founded in 2013 by the original creators of Apache Spark at UC Berkeley’s AMPLab. The founders open-sourced Spark and then… Continue reading on M

Databricks was founded in 2013 by the original creators of Apache Spark at UC Berkeley’s AMPLab. The founders open-sourced Spark and then… Continue reading on M

Databricks was founded in 2013 by the original creators of Apache Spark at UC Berkeley’s AMPLab. The founders open-sourced Spark and then… Continue reading on M

Atlan MCP transforms metadata into real-time control for Snowflake, Databricks, and AI agents Continue reading on Medium »

llm evaluation for data pipelines is the load-bearing correctness discipline of the 2026 data stack — the difference between a RAG chatbot… Continue reading on

A practitioner's guide to making your Iceberg data lake self-managing — from understanding why lakes degrade, through the control plane architecture, to choosin

The recommendation is clean, modern and sensible. It also assumes you do not have an Airflow instance, four hundred stored procedures, and a team that just lear

Subtitle: From “hope and pray” to “validate early, validate often” — how to catch bad data before it reaches your CEO’s dashboard. Continue reading on Medium »

The Agentic Data Operations Platform (ADOP) is a reference architecture on Amazon Bedrock that uses specialized AI agents to automate the full Bronze-to-Silver-

How to layer row-level quarantine at ingestion and YAML dataset contracts at the quality boundary — with alert routing, persisted failure artifacts, and CI en

Every enterprise data stack today looks broadly the same. At the bottom there’s a warehouse or lakehouse — Snowflake, Databricks, Redshift… Continue reading on

Andreessen Horowitz has two partners sitting on the boards of companies that now compete with each other: Ben Horowitz at Databricks and Martin Casado at Fivetr

Healthcare data platforms are evolving beyond ETL, using AI for anomaly detection, entity matching, forecasting, compliance, and data quality.

Migrating ~1000 stored procedures and ~100 SSIS jobs to dbt with Claude Code: equivalence testing, decomposition, context engineering. Continue reading on Mediu

Auto-termination and auto-suspend are idle timeouts, not cost strategies: they trim tails after the waste happened and say nothing about size, tier, or intent.

How the data lakehouse works in production — the internal mechanics, the data flow patterns, why...

The lakehouse community spent this week deciding what gets carried forward and what gets left behind....

On September 2025, Fivetran acquired Tobiko Data, the company behind SQLMesh and SQLGlot. A month...

Most production data pipelines do not fail because their underlying mathematical algorithms are flawed they implode because human domain… Continue reading on IL

How to Replicate MySQL to BigQuery with Sling Last updated: July 2026 Getting MySQL...

Integrate dbt tests, Great Expectations, and CI/CD to automate data quality checks across environments and scale with confidence.

Data Engineering, Lakehouse, Medallion, Streaming, Data Mesh, Data Vault & Modern Platform Architecture Continue reading on Medium »