2 posts tagged with "Pipelines"

Fresh gold tables are useless if the semantic model still points at yesterday, or if refresh burns an hour on a fact you could have incrementalized. Refresh is an architecture problem: orchestration, identity, and storage mode. Treat it as a portal checkbox and you invent a second process nobody can explain.

Figure 1. Direct Lake connects the semantic model to Delta in OneLake. Source: Microsoft Learn: Direct Lake overview.

Large facts can "succeed" on every run and still lie. Row counts look fine. Pipelines are green. Measures drift because deletes, late arrivals, and corrections were never part of the incremental contract. Analysts discover it weekly. Engineers call it flaky. It is usually a missing agreement.

Treat refresh drift as a contract break. Align watermark semantics with business expectations for corrections. Fast materialization without drift tests just fails faster. Publish a refresh SLA per fact grain. Make drift checks part of the pipeline, not a weekly analyst surprise. Sales-line style facts are the classic patient; the disease is mismatched notebooks, watermarks, and semantic consumers.

Figure 1. Green refreshes are not the same as honest facts. Contracts must cover late arrivals and corrections. Source: Microsoft Learn: data refresh in Power BI.