Tech Tutorials

Starting with a local Parquet file, then joining it to data stored in the cloud The post Building a Data Lakehouse with DuckDB and DuckLake appeared first on To

Every platform modernisation programme has the same first act, and it is never the interesting one. Before anyone gets to write a line of dbt , somebody has to

Every enterprise data stack today looks broadly the same. At the bottom there’s a warehouse or lakehouse — Snowflake, Databricks, Redshift… Continue reading on

The Batch Processing Hangover For decades, enterprise integration ran on a simple rhythm: extract, transform, load — repeat every night at 2 AM. Batch ETL pipel

TL;DR Continue reading on Medium »

Building a production-ready RSS pipeline with Python, Docker, PostgreSQL, and Kestra The post I Built My Second ETL Pipeline. This Time, I Started Thinking Like

In scenarios such as data migration, cross-cloud synchronization, and object storage backup, juicefs sync is commonly used to transfer large volumes of data. Wh

Your pipeline failed at 2am. What if it could fix itself? Continue reading on Data Engineer Things »

How Pandas chunking, Dask, and Polars help process millions of records when adding more compute isn't an option. The post What Can We Do When Memory Becomes the

A reflection on the first month of learning data engineering in public, and what actually kept me going. The post One Month Into Learning Data Engineering in Pu

A practical data engineering onboarding workflow for environment setup, automated testing, and AI-assisted development. The post Your First Task as a Data Engin

I tried to make my ETL pipeline production-ready. Three things broke. Each one taught me something scripting alone never could. The post I Thought Data Engineer

Rebuilding Postgres state in Snowflake from CDC logs, with live views, historical tables, and as-of reads for consistent pipelines. Continue reading on fresha-d

Author(s): Satish Kumar Originally published on Towards AI. Connections, Roles, and Warehouses: Getting CoCo Desktop Production-Ready from Day One Snowflake COC

By James Fahey Continue reading on Medium »

Current requirement from my client: Upload 3 Excel sheets One sheet contains 150k+ rows Another contains 40k+ rows Data needs to be inserted into multiple relat

How a single JAR on a laptop produced sub-millisecond internal pipeline latency while published Flink/Spark benchmarks show the real cost… Continue reading on M

From just HTTP fetch to full orchestration — without burning your weekend on infra you don’t need yet. Continue reading on JavaScript in Plain English »

ETL systems are uncompromisingly literal — and that is precisely why they age poorly. Continue reading on Medium »

A Decision-Tree Approach to Cracking Senior Data Engineering System Design Rounds Continue reading on Towards Data Engineering »

How I Combined Pandas, Dask, and SQLAlchemy for Speed and Reliability Continue reading on Top Python Libraries »

When I started working in data engineering, nothing felt simple. Even small tasks stretched longer than expected. Pipelines broke for… Continue reading on Mediu

Continue reading on Medium »

From raw data to real-time dashboards — a hands-on walkthrough of modern pipeline architecture using Kafka, Spark, dbt, and Airflow, plus… Continue reading on M