Skip to main content

One post tagged with "Data Engineering"

View All Tags

Keep Fabric thin: Airflow orchestrates, Fabric does the work

· 7 min read
Sai Prudhvi Neelakantam
Senior Consultant, Data Engineering & AI at Evidi

Fabric notebooks and pipelines are great at doing work. They are mediocre at being the calendar, the dependency graph, and the on-call story for everything around them. The moment three jobs must finish before a fourth can start, you need an orchestration plane that is not pretending to be another transform engine.

Airflow fits when you want an honest DAG: schedule, dependencies, retries, and a run history humans can read. Keep tasks thin. Call Fabric. Do not rebuild Fabric inside operators because it was convenient on a Friday.

Apache Airflow job canvas tile for running a Fabric item

Figure 1. Fabric Airflow jobs can trigger notebooks, pipelines, and other items from a DAG without stuffing transform logic into the orchestrator. Source: Microsoft Learn: run a Fabric item using Apache Airflow.