Legacy to lakehouse: semantic parity is the real migration

Copying tables into a lakehouse is the easy part. The hard part is whether 0 means zero or was null in the legacy system, and whether a 0.01 drift is rounding or a bug.

I treat migration as a semantic contract, not a file move. Stakeholders do not celebrate that Delta exists. They celebrate that last month's margin still matches within agreed rules.

Figure 1. Shortcuts and schemas change where data appears, not what it means. Source: Microsoft Learn: lakehouse schemas.

I have cut over Import and DirectQuery estates onto Fabric lakehouses where row counts looked perfect and finance still rejected the pilot. The bytes landed. The meaning did not. That gap is the real migration work.

The core idea

Parity is a written equivalence between legacy semantics and lakehouse semantics, enforced with automated compares, not a vibe that "the numbers look close."

Moving from classic Power BI Import or DirectQuery into a lakehouse-backed model changes engines, null handling, type coercion, and often the grain of intermediate tables. If you only validate that tables exist and schemas look familiar, you will discover measure drift in a steering meeting. Treat NULL versus 0, money precision, dates, flags, and measures as first-class contract clauses when reports are the acceptance surface.

Parity bugs that look fine in a demo

NULL versus 0. Legacy empty-string nulling patterns become integers that keep 0. Counts, averages, and joins shift. A filter that once excluded unknowns now includes zeros as real values. The reverse also happens: missing stored as 0 becomes NULL in silver, and "active" counts drop.

Float pennies. Unit costs that differ by 0.01 across rows explode into sum gaps you will debate for weeks. Import sometimes hid this via load or model rounding. Spark and Delta keep the float noise unless you decide decimal types and rounding early.

Datetime rebase modes. Spark legacy versus correct rebase settings change history silently if you do not pin them. A timestamp that looked fine in a sample month breaks around historical DST edges. Cluster-default notebooks are not reproducible migration tools.

Type once, early. Casting in gold while bronze stays stringy means every consumer re-learns the same bugs. DirectQuery often deferred typing to the engine; delayed lakehouse typing recreates that ambiguity with more hops.

Measure drift. Even when fact columns match, DAX that assumed Import cardinality, blank handling, or relationship behavior can disagree with Direct Lake or a re-modeled star. Parity is ETL plus the semantic model contract.

A contract approach that survives cutover

1. Define equivalence rules per column class

Write rules for money, quantities, flags, empty strings, dates, and identifiers. Example classes: "empty string in source becomes NULL in silver," "legacy 0 on nullable amount means unknown, not zero," "currency rounds half-up to 4 decimals before gold." Without classes, every table invents a private dialect.

2. Automate row and aggregate compares for P0 tables

Compare keys, null rates, distinct counts, and money sums between legacy extracts and lakehouse outputs. Row counts alone lie. Keep tolerances explicit: exact for keys and flags, bounded epsilon only where float is unavoidable and documented.

3. Pin Spark session settings in history-sensitive notebooks

Rebase modes, timezones, and parser options belong in the notebook or job definition, not in whoever last touched the cluster. Migration history is a compliance artifact. Silent default changes are defects.

4. Type as early as practical in the lineage

Promote decimals, dates, and booleans in silver when the rule is known. Gold should not be the first place a decimal exists if five notebooks already summed the string column differently. Early typing also makes Direct Lake and SQL endpoint consumers less surprising.

5. Include measures and relationships in the parity pack

For report cutovers, freeze a small suite of page-level and measure-level compares against a known legacy workspace. Document relationship differences (single versus both, missing members, blank row behavior). If Import used a calculated column that lakehouse pushes upstream, name that rewrite so nobody "fixes" it back in DAX during hypercare.

Figure 2. Moving content across stages does not prove semantic parity. Source: Microsoft Learn: deployment pipelines.

Failure modes I design against

Row counts match while money columns drift. The demo passes. Finance fails you.

Fixing tolerance in validation instead of fixing the write path. Widening epsilon until green is how you industrialize the bug.

One giant notebook cell that hides which table failed parity. Failures must name table, column class, and rule. Otherwise hypercare becomes archaeology.

Declaring ETL done before measure packs are compared. Stakeholders experience migration through reports. Meet them there.

Assuming DirectQuery blank behavior equals lakehouse NULL behavior. Engines disagree until you force the contract.

Shortcuts and schema moves treated as meaning-preserving. Location changed. Semantics still need proof.

Trade-offs

Real parity work slows the first cutover. You will argue whether legacy 0 was a business zero or a missing sentinel. That argument is cheaper before go-live than after. Strict typing early can break dirty bronze that used to "just load" into Import; that pain usually surfaces data quality while you still have rollback. Automated compares need stable legacy extracts and keys. Without reproducible snapshots, parity jobs go flaky and people stop trusting them.

What I would put on an ADR

1. Written NULL, empty, and zero rules by column class.
2. Parity jobs for P0 tables with named tolerances.
3. Explicit Spark rebase, timezone, and type policy in jobs that write history.
4. Where types are enforced in the medallion (bronze stay raw versus typed silver).
5. Measure and relationship compare pack for report cutover.
6. Definition of done that includes stakeholder sign-off on parity reports, not only pipeline green.

Primary references: Lakehouse schemas, Lakehouse overview, and Microsoft Learn guidance on Direct Lake and semantic models for Power BI cutovers.

Closing

If stakeholders argue about a 0.01, you do not have a reporting issue yet. You have an unfinished migration contract.

Land the files. Prove NULL, zero, money, dates, and measures under rules you can show in a review. Lakehouse migration succeeds when meaning survives the engine change, not when Delta folders look busy.