4 posts tagged with "Direct Lake"

Fresh gold tables are useless if the semantic model still points at yesterday, or if refresh burns an hour on a fact you could have incrementalized. Refresh is an architecture problem: orchestration, identity, and storage mode. Treat it as a portal checkbox and you invent a second process nobody can explain.

Figure 1. Direct Lake connects the semantic model to Delta in OneLake. Source: Microsoft Learn: Direct Lake overview.

Checklist reporting loves wide pivots. Teams often build them late: SQL OUTER APPLY, Import mode, captions invented in the report. Refresh time climbs. Direct Lake never gets a fair chance. The semantic model pays for a shape that should have been decided in gold.

Pivot once in gold. Broadcast joins plus a single groupBy/pivot in PySpark produce a self-contained fact: one row per reading with stable keys. Captions match the language users already see in PDFs. Materialize for Direct Lake. Stop paying Import tax for wide pivots that only exist because the warehouse view stayed nested.

Figure 1. Direct Lake wants report-ready flat facts, not nested checklist trees that only Import mode can fake. Source: Microsoft Learn: understand relationships.

Telemetry tables named after pipelines invite people to treat events like disposable logs. Reports then reinvent grain, matching, and captions in DAX. Volume pushes Import mode until refresh becomes the weekly outage.

Name the business concept (events). Enrich with the same matching logic used by service-device facts. Land them in gold for Direct Lake. Captions, query folders, and descriptions are part of the semantic handoff. Watermark shortcuts and clean views are part of the bronze-to-gold story, not footnotes.

Figure 1. IoT events are facts. Land them as modeled destinations, not as log piles that only Import mode can fake into a report. Source: Microsoft Learn: Eventstreams overview.

Parameterized DataLake expressions can work for months and then fail across stages overnight. Teams panic-rewrite convention docs as if literal OneLake URLs were always the design goal. That erases history and guarantees the workaround becomes religion.

Reframe "literal URL required" failures as service regressions. Ship validators and literal patches as temporary controls. Keep a Test to Preprod to Prod promotion story under the workaround. File and track the Microsoft support trail next to the repo convention. Do not pretend the platform always wanted hard-coded URLs.

Figure 1. Direct Lake on OneLake is the intended pattern. Literal URL patches are controls when parameters regress. Source: Microsoft Learn: Direct Lake storage.