Skip to main content

One post tagged with "Spark SQL"

View All Tags

Type each column once, as early as you dare

· 6 min read
Sai Prudhvi Neelakantam
Senior Consultant, Data Engineering & AI at Evidi

Re-CASTing the same physical column in every medallion layer feels careful. It is usually how drift multiplies. Bronze casts to decimal. Silver casts again "to be safe." Gold casts a third time and invents an unknown literal with yet another type. Consumers inherit a guessing game.

Type each published column once, as early as you dare (usually bronze, sometimes the first stable clean view). Silver and gold keep aliases, business calculations, and typed unknown-row literals only. UNION alignment casts are the exception, not the lifestyle. Lint the rest.

Configure column mapping in a Fabric pipeline

Figure 1. Type each column once, as early as you dare. Downstream layers should inherit the contract, not reinvent CASTs. Source: Microsoft Learn: copy data activity.