Workflow Orchestration
Schedule and monitor data pipelines with Airflow, Dagster, or Prefect.
0%
Confidence · no data yet
After this skill you can…
- Build a DAG in Airflow with sensors and operators
- Implement backfilling and retries
- Set up monitoring and SLA alerts
Prerequisites
Watch (10 videos)
Data Triggers and Advanced Control Flow With Lakeflow Jobs
→ Design workflows→ Implement conditional logic→ Use looping and job modularity
Dagster vs Airflow vs Prefect | Which Workflow Platform is BEST in 2025?
→ Orchestrate data workflows with Dagster→ Compare Airflow and Prefect for workflow management
AWS Dev Day - Data & Analytics | Amazon Managed Airflow or AWS Step functions? (Hebrew)
→ Orchestrate workflows with AWS Step Functions→ Use Amazon Managed Airflow for data pipelines
Orchestrating Spark Jobs with Kubeflow // Sadik Bakiu // MLOps Meetup #71
→ Orchestrate Spark jobs with Kubeflow→ Integrate Apache Spark with Kubernetes
Orchestrating Machine Learning Workflows with Prefect // Kevin Kho // MLOps Meetup #94
→ Build a Prefect workflow for ML pipelines→ Orchestrate a hyperparameter grid search with Dask
Unified orchestration for any workload with Lakeflow Jobs - Data Engineering with Databricks
→ Streamline data workflows→ Implement workflow automation
Orchestration With Lakeflow Jobs
→ Orchestrate data pipelines→ Manage workflows→ Automate tasks
Lakeflow In Action (Gourmet Pipeline Demo)
→ Automate workflow orchestration→ Manage data pipelines
Job Orchestration in Databricks | Data Engineering in Databricks
→ Orchestrate jobs in Databricks→ Automate data pipelines with Spark
Sponsored by: Astronomer | Unlocking the Future of Data Orchestration: Introducing Apache Airflow® 3
→ Design data workflows→ Implement workflow automation
Read (10 articles)
📄
DeepCamp AI