Skip to main content

One post tagged with "Data Engineering"

View All Tags

Keep Fabric thin: Airflow orchestrates, Fabric does the work

· 7 min read
Sai Prudhvi Neelakantam
Senior Consultant, Data Engineering and AI at Evidi

Fabric notebooks and pipelines are great at doing work. They are mediocre at being the calendar, the dependency graph, and the on-call story for everything around them. The moment three jobs must finish before a fourth can start, you need an orchestration plane that is not pretending to be another transform engine.

Airflow fits when you want an honest DAG: schedule, dependencies, retries, and a run history humans can read. Keep tasks thin. Call Fabric. Do not rebuild Fabric inside operators because it was convenient on a Friday.

Apache Airflow job canvas tile for running a Fabric item

Figure 1. Fabric Airflow jobs can trigger notebooks, pipelines, and other items from a DAG without stuffing transform logic into the orchestrator. Source: Microsoft Learn: run a Fabric item using Apache Airflow.