Data Engineering

In this project, I created an updated version of my previous project. Although I didn’t use them in the previous one, this time I used… Continue reading on Medi

Data has become the driving force behind almost every modern business. Whether you’re shopping online, scrolling through social media… Continue reading on Mediu

windmill is the pick-one architectural decision that finally collapses "we need an orchestrator and an internal-tool builder and a secret… Continue reading on M

Building a production-ready RSS pipeline with Python, Docker, PostgreSQL, and Kestra The post I Built My Second ETL Pipeline. This Time, I Started Thinking Like

In scenarios such as data migration, cross-cloud synchronization, and object storage backup, juicefs sync is commonly used to transfer large volumes of data. Wh

Your pipeline failed at 2am. What if it could fix itself? Continue reading on Data Engineer Things »

How Pandas chunking, Dask, and Polars help process millions of records when adding more compute isn't an option. The post What Can We Do When Memory Becomes the

Outgrowing Ponder? This guide walks through migrating your indexer to Envio HyperIndex — keeping your...

Introduction Modern data pipelines are designed to process data continuously, whether hourly, daily,...

How I designed and built an end-to-end data engineering system that any team could inherit, run, and extend — using Python, DuckDB, Apache… Continue reading on

From Experience to Understanding: What Two Databricks Learning Tracks Taught Me About the Modern Data Lifecycle Continue reading on Medium »

Learn how to write a US-data-engineering cover letter from an Indian IT services background. Includes a concrete before/after bullet rewrite and a copy-paste ch

How a simple command can save hours of debugging, reduce CI failures, and improve your team’s code quality. Continue reading on Medium »

How a simple command can save hours of debugging, reduce CI failures, and improve your team’s code quality. Continue reading on Medium »

Building ossfind.com's data pipeline: two-phase seeding, GitHub API fetching, Claude Haiku summaries, and the DO NOTHING bug that silently erased polished conte

If you manage a ClickHouse cluster in production, you may have hit duplicate rows or the "too many...

Designing a production-ready data ingestion pipeline with validation, analytics, testing, and cloud deployment. Continue reading on Medium »

The moment I stopped worrying about the destination and started appreciating the journey Continue reading on Medium »

Build your first Python Data pipeline using the News API, and make some sense of the...

A reflection on the first month of learning data engineering in public, and what actually kept me going. The post One Month Into Learning Data Engineering in Pu

Apache Iceberg looked like the answer to everything when we first adopted it. Open format, ACID...

A practical data engineering onboarding workflow for environment setup, automated testing, and AI-assisted development. The post Your First Task as a Data Engin

Three months into my first role, I found an uncomfortable reality: I didn’t really control the data I depended on daily. A data engineer… Continue reading on Me

Traditional batch-first data pipelines introduce artificial delays in data availability, forcing enterprise decisions to be made on stale information. This arti

Take-homes grew from 4 hours to 20. No pay, no feedback, AI banned with no rubric updates. The DE interview is now just unpaid consulting.

As ClickHouse® deployments grow beyond a single server, ensuring high availability, scalability, and...

Apache Airflow has become one of the most widely used workflow orchestration platforms for building,...

If you’re getting into data engineering, you’ve probably heard the term “Airflow” thrown around a lot. Continue reading on Medium »

For years, data engineering and software engineering have lived in parallel worlds. Application developers shipped code through structured… Continue reading on

The Chevron Microsoft gas deal hands the oil major a 20-year contract to power a giant West Texas data centre. For a company that built its AI plans on renewabl

Why I chose this topic: In my first year as a junior engineer, I pushed a non-idempotent job that...

Every data engineer who works across platforms knows this pain: You build a clean ingestion layer...

In a distributed analytical gbase database, many performance issues are baked in at the table design...

Legacy ETL modernization is often described as a technology migration. Move DataStage jobs to...

There is a quiet absurdity at the center of most data work, and once you notice it you cannot stop...

I offer 1:1 mentoring and interview prep for data engineers on Topmate. Now let’s get into it. Continue reading on Medium »

The AWS Data Engineer Associate is one of the newer associate-level certs, and because it's new, the...

Vinay Chella and Akshat Goel discuss the challenges of running traditional CDC across heterogeneous databases during peak order traffic. They explain how Debezi

12 real pipeline architecture interview questions with answers: batch vs streaming, idempotency, backfills, DAGs, schema evolution, and monitoring.

One of the reasons ClickHouse delivers exceptional analytical performance is its ability to optimize...

Distributed databases represent one of the most significant architectural paradigms in contemporary data engineering. After nine years… Continue reading on Medi

How a 17-feed RSS pipeline stays alive when TOI, NDTV, Moneycontrol, The Wire, and Scroll.in all break differently. Fault-tolerance patterns, source quality sco

How Flink uses checkpoint barriers to take consistent snapshots of your running job so a crashed pod becomes a minor blip, not a… Continue reading on Towards Da

So, yes. Today's goal is to get the 30hr SQL Bootcamp completed (or at least as much as I can), I am...

Why we’re open-sourcing State-Aware Orchestration for dbt in Orchestra and dbt core Continue reading on Medium »

Original Japanese article: Iceberg × Glue Data Catalogのスキーマ変更履歴をaws glue get-table-versionsで確認する ...

GBase 8a, as an MPP analytical database, does not use WAL transaction logs. Instead, it relies on the...

A practical build-in-public note on automating repetitive data engineering artifacts from source-to-target mapping metadata.