Data Engineering Reads

In this project, I created an updated version of my previous project. Although I didn’t use them in the previous one, this time I used… Continue reading on Medi

Data has become the driving force behind almost every modern business. Whether you’re shopping online, scrolling through social media… Continue reading on Mediu

windmill is the pick-one architectural decision that finally collapses "we need an orchestrator and an internal-tool builder and a secret… Continue reading on M

Building a production-ready RSS pipeline with Python, Docker, PostgreSQL, and Kestra The post I Built My Second ETL Pipeline. This Time, I Started Thinking Like

In scenarios such as data migration, cross-cloud synchronization, and object storage backup, juicefs sync is commonly used to transfer large volumes of data. Wh

Your pipeline failed at 2am. What if it could fix itself? Continue reading on Data Engineer Things »

How Pandas chunking, Dask, and Polars help process millions of records when adding more compute isn't an option. The post What Can We Do When Memory Becomes the

Outgrowing Ponder? This guide walks through migrating your indexer to Envio HyperIndex — keeping your...

Introduction Modern data pipelines are designed to process data continuously, whether hourly, daily,...

How I designed and built an end-to-end data engineering system that any team could inherit, run, and extend — using Python, DuckDB, Apache… Continue reading on

From Experience to Understanding: What Two Databricks Learning Tracks Taught Me About the Modern Data Lifecycle Continue reading on Medium »

Learn how to write a US-data-engineering cover letter from an Indian IT services background. Includes a concrete before/after bullet rewrite and a copy-paste ch

How a simple command can save hours of debugging, reduce CI failures, and improve your team’s code quality. Continue reading on Medium »

How a simple command can save hours of debugging, reduce CI failures, and improve your team’s code quality. Continue reading on Medium »

Building ossfind.com's data pipeline: two-phase seeding, GitHub API fetching, Claude Haiku summaries, and the DO NOTHING bug that silently erased polished conte

If you manage a ClickHouse cluster in production, you may have hit duplicate rows or the "too many...

Designing a production-ready data ingestion pipeline with validation, analytics, testing, and cloud deployment. Continue reading on Medium »

The moment I stopped worrying about the destination and started appreciating the journey Continue reading on Medium »

Build your first Python Data pipeline using the News API, and make some sense of the...

A reflection on the first month of learning data engineering in public, and what actually kept me going. The post One Month Into Learning Data Engineering in Pu

Apache Iceberg looked like the answer to everything when we first adopted it. Open format, ACID...

A practical data engineering onboarding workflow for environment setup, automated testing, and AI-assisted development. The post Your First Task as a Data Engin

Three months into my first role, I found an uncomfortable reality: I didn’t really control the data I depended on daily. A data engineer… Continue reading on Me

Traditional batch-first data pipelines introduce artificial delays in data availability, forcing enterprise decisions to be made on stale information. This arti