Skip to main content

One post tagged with "PySpark"

View All Tags

Pivot once in gold: measurable controls as a flat fact

· 6 min read
Sai Prudhvi Neelakantam
Senior Consultant, Data Engineering & AI at Evidi

Checklist reporting loves wide pivots. Teams often build them late: SQL OUTER APPLY, Import mode, captions invented in the report. Refresh time climbs. Direct Lake never gets a fair chance. The semantic model pays for a shape that should have been decided in gold.

Pivot once in gold. Broadcast joins plus a single groupBy/pivot in PySpark produce a self-contained fact: one row per reading with stable keys. Captions match the language users already see in PDFs. Materialize for Direct Lake. Stop paying Import tax for wide pivots that only exist because the warehouse view stayed nested.

Active versus inactive relationships in a model diagram

Figure 1. Direct Lake wants report-ready flat facts, not nested checklist trees that only Import mode can fake. Source: Microsoft Learn: understand relationships.