Keep Fabric thin: Airflow orchestrates, Fabric does the work
Fabric notebooks and pipelines are great at doing work. They are mediocre at being the calendar, the dependency graph, and the on-call story for everything around them. The moment three jobs must finish before a fourth can start, you need an orchestration plane that is not pretending to be another transform engine.
Airflow fits when you want an honest DAG: schedule, dependencies, retries, and a run history humans can read. Keep tasks thin. Call Fabric. Do not rebuild Fabric inside operators because it was convenient on a Friday.
Figure 1. Fabric Airflow jobs can trigger notebooks, pipelines, and other items from a DAG without stuffing transform logic into the orchestrator. Source: Microsoft Learn: run a Fabric item using Apache Airflow.
I learned this the hard way on Fabric estates that looked fine in the portal and fell apart the first time three jobs needed to finish before a fourth could start. The workload logic belonged in Fabric. The coordination did not. Collapsing those concerns into one place made every change feel riskier than it should.
The core idea
Keep the workload logic in Fabric. Keep orchestration in Airflow. Make each task do one thing well.
Fabric still owns notebooks, pipelines, lakehouse tasks, warehouse operations, and semantic refresh. Airflow owns timing, dependencies, retries, alerting, and the operational story of the run. That separation keeps pipelines easier to test, easier to recover, and much easier to extend when a new dataset or downstream consumer appears.
Figure 2. Monitoring belongs with the orchestration contract, not as a private notebook side effect. Source: Microsoft Learn: run a Fabric item using Apache Airflow.
Airflow is not a second Fabric. If you start reimplementing transforms inside operators because it was convenient, you have created a second source of truth. Convenience is how orchestration layers go thick and then opaque.
A model that stays explainable
1. One DAG, thin tasks, honest names
Map the business flow to a DAG that a new hire can read without a walkthrough: land, validate, transform, publish, notify. Each task should do one clear thing. If a task grows into everything after bronze, you lose clean retries and you lose the ability to say which step actually failed. Prefer many small operators that call Fabric over one giant operator that hides a private workflow.
2. Fabric executes; Airflow coordinates
Trigger Fabric jobs, pipelines, or notebooks from Airflow. Do not copy the Spark logic into Airflow Python for speed. The execution plane stays in Fabric where capacity, identities, and lakehouse bindings already live. Airflow should pass parameters, wait for completion, and react to status. When status is ambiguous, fix the Fabric job contract. Do not paper over it with longer sleeps.
3. Explicit success and failure contracts
Every task needs a definition of done that is checkable: row counts, file presence, quality gate pass, pipeline status, semantic refresh state. It ran is not a contract. Partial success without a named gate is how gold tables get refreshed from incomplete silver. Put the gate in the DAG where it is visible, not only in a notebook comment.
4. Retries that match the failure type
Transient platform blips deserve automatic retry with backoff. Bad data and broken schemas do not. Treat validation failures as terminal for that run unless you have a deliberate quarantine path. Blind retries on quality failures burn capacity and hide the real incident under a green-looking retry storm.
5. Observability that names the step
Operators, run ids, and alert text should point at the failing Fabric artifact and the DAG task in the same message. A Slack ping that says pipeline failed without the task id and the Fabric run url forces people to reconstruct context under pressure. Design the alert as part of the DAG, not as an afterthought in a shared mailbox rule.
6. Environments as parameters, not forks
Dev, Test, and Prod should be the same DAG shape with different Fabric targets, identities, and schedules. Copy-pasted DAGs per environment drift on the first hotfix. Parameterize workspace, lakehouse, and connection ids the same way you would parameterize a Fabric deploy. Pretty names change. Ids should be the contract.
Failure modes I design against
Orchestrator does the transform. Airflow notebooks or Python operators slowly recreate what Fabric already owns. Now you have two places to patch logic and no single owner.
One fat task for the whole medallion path. Failure at gold forces a full rerun from bronze. Time and cost explode. Partial recovery becomes folklore.
Hidden dependencies outside the DAG. A semantic model refresh that someone runs after lunch is still a dependency. If it is not in the graph, on-call will learn about it from a broken report, not from Airflow.
Success means HTTP 200 from a trigger. The trigger accepted the job. The job later failed. Your DAG already moved on. Wait for terminal state, or you will publish lies.
Secrets and identities living in two places. Airflow connections and Fabric workspace identities disagree. Runs work on one laptop schedule and fail in the shared environment. Centralize how service principals and key vault references are bound.
Trade-offs
Airflow adds another runtime to operate: workers, schedulers, package versions, and network paths into Fabric APIs. That cost is real. For a single nightly notebook with no dependents, Fabric native scheduling may be enough. Reach for Airflow when cross-job dependencies, shared SLAs, or multi-system coordination show up.
You will also negotiate ownership. Platform teams often want Fabric-native orchestration. Data engineers often want DAG readability and ecosystem operators. Pick based on the dependency graph you must explain, not on which demo looked smoother in a workshop.
API throttling matters when many DAGs poke the same Fabric tenant. Set concurrency limits before you scale hourly jobs.
What I would put on an ADR
- Fabric owns data plane work; Airflow owns schedule, dependency, retry, and alert contracts.
- Tasks are single-purpose and named for the business step, not for the tool call.
- Terminal state of the Fabric job is the task success condition, not trigger acceptance.
- Quality gates are explicit DAG tasks with defined fail behavior.
- Environment targets are parameterized; no forked DAGs per stage.
- Alerts include DAG task id, Fabric run url, and the artifact that failed.
- No business transform logic duplicated into Airflow operators without a written exception.
Primary references: Microsoft Learn on Fabric Data Factory and pipelines, notebooks in Fabric, and Apache Airflow docs on DAGs and tasks.
Closing
Orchestration fails in boring ways. Not because Fabric cannot run a notebook, but because we pretended timing and dependency were optional features you could bolt on later with a calendar reminder.
If your architecture diagram has one box labeled jobs, redraw it.
Workload. Schedule. Contract. Three concerns. Two tools. One readable DAG.