Data
Data Engineering
ETL pipelines, data warehousing, streaming, orchestration and lakehouse architecture
Skills in this topic
5 skills — Sign in to track your progress
ETL Basics
beginner
Write a Python ETL pipeline with pandas
Workflow Orchestration
intermediate
Build a DAG in Airflow with sensors and operators
Streaming Data
intermediate
Produce and consume Kafka topics with Python
Data Warehousing
intermediate
Model a star schema with dbt
Lakehouse Architecture
advanced
Manage ACID transactions on a data lake with Delta Lake
Medium · Python
🔄 Data Engineering
⚡ AI Lesson
3d ago
Data Engineering ETL Project
In this project, I created an updated version of my previous project. Although I didn’t use them in the previous one, this time I used… Continue reading on Medi
Medium · Python
🔄 Data Engineering
⚡ AI Lesson
4d ago
Data Engineering: A Simple Guide to Building a Career in the World of Data
Data has become the driving force behind almost every modern business. Whether you’re shopping online, scrolling through social media… Continue reading on Mediu
Medium · Python
🔄 Data Engineering
⚡ AI Lesson
1w ago
Windmill for Data Engineering: TypeScript/Python Scripts, Flows & Self-Hosted OSS
windmill is the pick-one architectural decision that finally collapses "we need an orchestrator and an internal-tool builder and a secret… Continue reading on M
Medium · Python
🔄 Data Engineering
⚡ AI Lesson
1mo ago
Building a Production-Style Weather Analytics Pipeline from Scratch: ETL, ELT, Star Schema, and…
How I designed and built an end-to-end data engineering system that any team could inherit, run, and extend — using Python, DuckDB, Apache… Continue reading on
Medium · Data Science
🔄 Data Engineering
⚡ AI Lesson
1mo ago
From Experience to Understanding: What Two Databricks Learning Tracks Taught Me About the Modern…
From Experience to Understanding: What Two Databricks Learning Tracks Taught Me About the Modern Data Lifecycle Continue reading on Medium »
Medium · Data Science
🔄 Data Engineering
⚡ AI Lesson
1mo ago
Why Every Data Engineer Should Run poe all Before Every Git Push
How a simple command can save hours of debugging, reduce CI failures, and improve your team’s code quality. Continue reading on Medium »
Medium · Python
🔄 Data Engineering
⚡ AI Lesson
1mo ago
Why Every Data Engineer Should Run poe all Before Every Git Push
How a simple command can save hours of debugging, reduce CI failures, and improve your team’s code quality. Continue reading on Medium »
Medium · Python
🔄 Data Engineering
⚡ AI Lesson
1mo ago
Building an End-to-End Data Engineering API with FastAPI, PostgreSQL, Docker and Google Cloud
Designing a production-ready data ingestion pipeline with validation, analytics, testing, and cloud deployment. Continue reading on Medium »
Medium · Data Science
🔄 Data Engineering
⚡ AI Lesson
1mo ago
Learning Databricks Has Taught Me to Enjoy the Process
The moment I stopped worrying about the destination and started appreciating the journey Continue reading on Medium »
Medium · Data Science
🔄 Data Engineering
⚡ AI Lesson
1mo ago
The Data Pipeline Hell: Why Most Data Scientists Don’t Control Their Data
Three months into my first role, I found an uncomfortable reality: I didn’t really control the data I depended on daily. A data engineer… Continue reading on Me
Hackernoon
🔄 Data Engineering
⚡ AI Lesson
1mo ago
Eliminating Data Latency with Event-Driven Pipelines at Enterprise Scale
Traditional batch-first data pipelines introduce artificial delays in data availability, forcing enterprise decisions to be made on stale information. This arti
Medium · Python
🔄 Data Engineering
⚡ AI Lesson
1mo ago
Why Apache Airflow is a Game Changer for Data Engineering
If you’re getting into data engineering, you’ve probably heard the term “Airflow” thrown around a lot. Continue reading on Medium »
Medium · DevOps
🔄 Data Engineering
⚡ AI Lesson
1mo ago
Why Data Engineers Can No Longer Afford to Ignore DevOps Practices
For years, data engineering and software engineering have lived in parallel worlds. Application developers shipped code through structured… Continue reading on
Medium · Data Science
🔄 Data Engineering
⚡ AI Lesson
1mo ago
System Design for Data Engineers: A Complete Guide (with Real Walkthroughs)
I offer 1:1 mentoring and interview prep for data engineers on Topmate. Now let’s get into it. Continue reading on Medium »
InfoQ AI/ML
🔄 Data Engineering
⚡ AI Lesson
1mo ago
Presentation: Write-Ahead Intent Log: A Foundation for Efficient CDC at Scale
Vinay Chella and Akshat Goel discuss the challenges of running traditional CDC across heterogeneous databases during peak order traffic. They explain how Debezi
Medium · Data Science
🔄 Data Engineering
⚡ AI Lesson
1mo ago
Day 28 of 32 Days of SQL Concepts — Distributed Databases
Distributed databases represent one of the most significant architectural paradigms in contemporary data engineering. After nine years… Continue reading on Medi
Medium · Data Science
🔄 Data Engineering
⚡ AI Lesson
1mo ago
Flink Checkpointing: How your job survives a crash
How Flink uses checkpoint barriers to take consistent snapshots of your running job so a crashed pod becomes a minor blip, not a… Continue reading on Towards Da
Medium · Data Science
🔄 Data Engineering
⚡ AI Lesson
1mo ago
Open-Sourcing State-Aware Orchestration (SAO) for dbt core
Why we’re open-sourcing State-Aware Orchestration for dbt in Orchestra and dbt core Continue reading on Medium »
Medium · Machine Learning
🔄 Data Engineering
⚡ AI Lesson
1mo ago
10 SQL Interview Problems That Tripped Me Up
The exact patterns that decide most data engineering SQL screens, with explanations that build real intuition. Continue reading on Medium »
Medium · Data Science
🔄 Data Engineering
⚡ AI Lesson
1mo ago
Open Table Format Explained: Features, Benefits, and Why It Matters in Modern Data Engineering
Initially, organizations relied heavily on data warehouses, storing structured data in traditional databases like MySQL and building… Continue reading on Medium
Medium · Python
🔄 Data Engineering
⚡ AI Lesson
1mo ago
Your Scraper Re-Downloads Everything. Most Didn’t Change.
Your scheduled scraper re-downloaded the whole corpus last night. A few thousand records. About forty of them actually changed since the… Continue reading on Me
Medium · Python
🔄 Data Engineering
⚡ AI Lesson
1mo ago
You’re Moving Data You Never Had to Move.
Five Snowflake patterns that quietly kill your external ETL — and the gotchas nobody warns you about. Continue reading on Medium »
InfoQ AI/ML
🔄 Data Engineering
⚡ AI Lesson
1mo ago
Cloudflare Identifies Query Planning Bottleneck in ClickHouse
Cloudflare recently described how a slowdown in its billing pipeline was traced to contention inside the query planning stage of ClickHouse. The team profiled t
Medium · Python
🔄 Data Engineering
⚡ AI Lesson
1mo ago
How I used Prefect for local pipeline orchestration on Windows with minimal setup
Workflow orchestration is one of the core requirements in almost every data engineering role. Continue reading on Medium »
Medium · Python
🔄 Data Engineering
⚡ AI Lesson
2mo ago
The freshness problem in Data Pipelines, and How to solve it in Airflow
Silent data failures are the worst kind. Here’s a system that makes them impossible Continue reading on Medium »
Medium · Data Science
🔄 Data Engineering
⚡ AI Lesson
2mo ago
I Rebuilt Our Entire Data Pipeline in Azure —Here's What I’d Do Differently
Not because it broke. Because it became impossible to change. Six months, pipelines, and a few mistakes we made twice. Continue reading on Towards Data Engineer
Medium · Python
🔄 Data Engineering
⚡ AI Lesson
2mo ago
Creating ADLS Gen2 Shortcuts in Microsoft Fabric via REST API — What the Docs Don't Tell You
A field guide for data engineers who've hit the 404/EntityNotFound wall Continue reading on Medium »
Medium · Machine Learning
🔄 Data Engineering
⚡ AI Lesson
2mo ago
Data Engineering for Machine Learning: From Basics to Advanced (With Python Examples)
Modern AI systems are powered by data, but raw data alone is useless without proper engineering. Continue reading on Stackademic »
Medium · Data Science
🔄 Data Engineering
⚡ AI Lesson
2mo ago
Data Engineering Project Layers
In data engineering, breaking down a project into Layers (also called Phases or Stages) allows you to isolate responsibilities, scale… Continue reading on Mediu
Medium · Data Science
🔄 Data Engineering
⚡ AI Lesson
2mo ago
7 Data Engineering Shifts Nobody’s Talking About Yet
The Hidden Blueprint for 10x Career Growth & Unstoppable Innovation by 2026 Continue reading on AI Simplified in Plain English »
Medium · Data Science
🔄 Data Engineering
⚡ AI Lesson
2mo ago
Why No One Understands Your Data Platform But You
A structured approach to documenting architecture for every audience Continue reading on Data Engineer Things »
Medium · Deep Learning
🔄 Data Engineering
⚡ AI Lesson
2mo ago
5 Data Engineering Problems I See In Almost Every Production System
After seeing these data engineering problems repeatedly across production systems, I would encourage every data engineer to understand how… Continue reading on
Medium · Machine Learning
🔄 Data Engineering
⚡ AI Lesson
2mo ago
Data Engineering Terminology Made Easy for Data Scientists, ML Engineers, and Anyone Who’s Ever…
Nobody Told You What Any of This Means in simple language — Until Now Continue reading on Medium »
Medium · Data Science
🔄 Data Engineering
⚡ AI Lesson
2mo ago
Databricks Lake base Evolution — Mandatory Shift from Provisioned Instances to Elastic Autoscaling
The OLTP landscape within the Data Intelligence Platform is undergoing an essential evolutionary step. Continue reading on Medium »
Medium · Data Science
🔄 Data Engineering
⚡ AI Lesson
2mo ago
How Snowflake Fixed the Biggest Problem in Data Engineering
A practical look at why modern teams are abandoning traditional database limitations. Continue reading on Medium »
Medium · Data Science
🔄 Data Engineering
⚡ AI Lesson
2mo ago
The Origins of Data Engineering: From Traditional ETL to AI-Ready Architectures
The origins of data engineering date back to the early enterprise data warehouse era of the 1980s and 1990s. During this period… Continue reading on Medium »
Medium · Machine Learning
🔄 Data Engineering
⚡ AI Lesson
2mo ago
From Experimental Notebooks to Production: A Data Engineer’s perspective of Scaling Data Science…
Originally published at harishkesavarao.github.io Continue reading on Medium »
Medium · Python
🔄 Data Engineering
⚡ AI Lesson
2mo ago
How I Built a Self-Updating Cloud Data Lake for $0 Using Python and GitHub Actions
A step-by-step guide to engineering a production-grade, hands-free ETL pipeline with Change-Data-Capture (CDC) logic. Continue reading on Medium »
KDnuggets
🔄 Data Engineering
⚡ AI Lesson
2mo ago
Top 10 Python Libraries for Data Engineering in 2026
Want to level up your data engineering toolkit? Here are some Python libraries that'll make your pipelines faster, cleaner, and easier to maintain.
Medium · Data Science
🔄 Data Engineering
⚡ AI Lesson
2mo ago
Apache Kafka & Spark Made Easy in a Real Hospital Data Architecture- Day 5
Day 5 of 6 — Complete System Design: From Architecture to Production Numbers Continue reading on Medium »
Medium · Python
🔄 Data Engineering
⚡ AI Lesson
2mo ago
How I Built an AWS Data Engineering Smoke Test with ECS, S3, and Athena
A walkthrough of how I built a synthetic data ingestion engine with Python, Codespaces, and AWS, why I did it, and key lessons learned Continue reading on Level
Medium · Data Science
🔄 Data Engineering
⚡ AI Lesson
2mo ago
Apache Kafka & Spark Made Easy in a Real Hospital Data Architecture-Day 3
Day 3 of 6 — Schemas, Failures, Retries & Designing Systems That Don’t Break Continue reading on Medium »
Medium · Machine Learning
🔄 Data Engineering
⚡ AI Lesson
2mo ago
What I Learnt Implementing a Medallion Architecture from Scratch on Databricks Using Washington…
9 hard-won lessons from a real end-to-end lakehouse build, EVLytics Continue reading on Towards Data Engineering »
Medium · Data Science
🔄 Data Engineering
⚡ AI Lesson
2mo ago
What I Learnt Implementing a Medallion Architecture from Scratch on Databricks Using Washington…
9 hard-won lessons from a real end-to-end lakehouse build, EVLytics Continue reading on Towards Data Engineering »
Medium · Data Science
🔄 Data Engineering
⚡ AI Lesson
2mo ago
The Complete Framework to Design ETL Pipelines in Interviews
A Decision-Tree Approach to Cracking Senior Data Engineering System Design Rounds Continue reading on Towards Data Engineering »
Medium · Python
🔄 Data Engineering
⚡ AI Lesson
2mo ago
The Hidden Complexity of Data Engineering in Regulated Industries (And What It Taught Me About…
Strict data formats and compliance requirements teach you more about clean software design than any course. Here is what working in a… Continue reading on Level
Medium · Python
🔄 Data Engineering
⚡ AI Lesson
3mo ago
I Stopped Fixing Broken Parsers at 3 AM , Here’s How We Outsourced Our DOM Extraction
It’s 3:00 AM on a Tuesday. Your PagerDuty alert is ringing. Continue reading on Medium »
Medium · DevOps
🔄 Data Engineering
⚡ AI Lesson
3mo ago
Modernizing Data Ingestion: An Async PostgreSQL Pipeline with Psycopg 3
Orchestrating high-performance migrations using asynchronous architectures and memory-safe processing. Continue reading on Medium »
DeepCamp AI