Semantic refresh architecture: notebooks, pipelines, and Direct Lake

Fresh gold tables are useless if the semantic model still points at yesterday, or if refresh burns an hour on a fact you could have incrementalized. Refresh is an architecture problem: orchestration, identity, and storage mode. Treat it as a portal checkbox and you invent a second process nobody can explain.

Figure 1. Direct Lake connects the semantic model to Delta in OneLake. Source: Microsoft Learn: Direct Lake overview.

I learned this on Fabric lakehouse stacks where bronze and silver looked healthy while reports still told last night's story. Transforms were fine. The semantic refresh contract was not.

The core idea

Building curated tables, refreshing the semantic model, and choosing how the model reads data are three different levers. Fixing one does not fix the others. Mixing them produces slow facts, surprise fallbacks, and overnight jobs that succeed while reports stay stale.

Direct Lake changes the freshness conversation. When the model reads Delta in OneLake without a classic Import copy, refresh stops meaning "reload every row into VertiPaq." It means the lake is correct, framing is current, and the connection identity can see the files. Import and hybrid shapes still need scheduled or triggered model refresh. Skipping checks on framing, MLVs, and connection policy is false confidence.

Three levers people mix up

1. Building curated tables. Notebooks, pipelines, Spark jobs, and materialized lake views write or maintain Delta.
2. Refreshing the semantic model. Schedule, on-demand, or post-pipeline trigger.
3. How the model reads data. Import, DirectQuery, Direct Lake, or SQL endpoint views. This decides how expensive (2) becomes.

A successful notebook run is not a successful semantic refresh. Direct Lake is not a free pass to ignore model operations. Document all three or you debug the wrong layer under pressure.

A model that stays explainable

1. Orchestrate build, then refresh

Pipelines should finish lakehouse transforms, validate outputs when you care about parity, then trigger model refresh only when something the model depends on changed. Blind nightly full refresh is a default, not a design. For Import tables, that trigger still matters. For Direct Lake heavy facts, land correct Delta and optionally nudge framing rather than reloading history. If gold did not move, do not burn capacity pretending it did.

2. Prefer Direct Lake for large facts when the layout fits

If a heavy fact is read through a SQL endpoint view because it worked in the lab, ask whether Direct Lake on the Delta table is the intended path. Wrong mode shows up as slow queries, odd refresh behavior, and capacity burn that looks like a DAX problem. Direct Lake wants clean Delta, sensible types, and a model shaped for that mode. SQL views in front of Direct Lake often reintroduce the latency you left Import to escape. Mixing Import dimensions with Direct Lake facts can be intentional; without a written reason it becomes tribal knowledge.

3. Treat Materialized Lake Views as their own freshness class

MLVs are not ordinary gold tables with a nicer name. They have their own refresh and dependency story. If a semantic model points at an MLV or at gold that depends on one, your pipeline DAG must include that refresh path. Hoping the view "is just SQL" yields correct notebooks and wrong reports. Write down who refreshes the MLV, what fails open versus fails closed, and whether the model can see a partially updated graph. Partial freshness is worse than honest staleness.

4. Identity for refresh must match the connection

Interactive SSO that works in the browser is not the same as a service principal or managed identity refreshing overnight. Design the connection the way production will authenticate. Microsoft docs strongly recommend a fixed identity for Direct Lake cloud connections. Personal accounts fail on vacation, Conditional Access, or MFA. Map workspace roles, lakehouse permissions, and the model connection to that same identity. "It works when I click Refresh" is not an operations proof.

Figure 2. Lakehouse as the analytical surface. Source: Microsoft Learn: lakehouse overview.

5. Know when classic refresh still matters

Direct Lake does not delete the need for refresh architecture. Import tables still need it. Composite models still need a clear story per table. Framing and metadata operations still belong in the runbook. Calculated tables and hybrid patterns can reintroduce load work even when the big fact is Direct Lake. Keep incremental discipline for Import: full materialization of a huge fact every run will eventually own your capacity calendar.

Failure modes I design against

Pipeline green, model stale. Transforms finished; nobody triggered the model or framing step.

Independent schedules. Model refresh ignores pipeline success, so you refresh incomplete gold or skip after a late land.

SQL endpoint comfort food. Heavy facts stay on views because prototyping was easy. Production pays in query time and confused Direct Lake expectations.

MLV ignored in the DAG. Definition exists; refresh ownership does not. Downstream gold and semantics drift on quiet nights.

Interactive identity for unattended work. Works on the author's session. Fails overnight under a different auth path.

Mixed modes without a legend. Nobody can say why dim is Import and fact is Direct Lake, so the next change guesses.

Trade-offs

Tighter orchestration means more pipeline authoring and more alerts. You will negotiate with capacity owners about burning CU on full Import refresh versus Direct Lake framing. Direct Lake reduces classic refresh pain for large facts but raises the bar on Delta hygiene, permissions, and model design. MLVs add elegance and dependency debt: every materialized hop needs a DAG owner, or freshness becomes folklore.

What I would put on an ADR

1. What builds tables versus what refreshes or frames models, with owners.
2. Storage mode per table or model, with a one-line reason.
3. Identity used for unattended Direct Lake and Import refresh.
4. Incremental or skip-if-unchanged policy for Import and lake builds.
5. MLV and dependent-gold refresh placement in the pipeline.
6. Alerting when the pipeline succeeds and semantic freshness checks fail.

Primary references: Direct Lake overview, Lakehouse overview, and Microsoft Learn on semantic model refresh and Direct Lake connections.

Closing

Semantic freshness is a platform contract. Treat it like deploy, not like a checkbox in the portal.

Build the lake. Make the model honest about how it reads that lake. Prove the overnight identity can do both without you watching. If your diagram has one arrow labeled Refresh, split build, mode, and model operations across owners.