5 posts tagged with "Data Quality"

Copying tables into a lakehouse is the easy part. The hard part is whether 0 means zero or was null in the legacy system, and whether a 0.01 drift is rounding or a bug.

I treat migration as a semantic contract, not a file move. Stakeholders do not celebrate that Delta exists. They celebrate that last month's margin still matches within agreed rules.

Figure 1. Shortcuts and schemas change where data appears, not what it means. Source: Microsoft Learn: lakehouse schemas.

Auto-selecting every AI-analyzed row feels helpful. It is how bad master data enters production with a smile. Users trust the model highlight. They click Import. Months later someone asks why an unknown code became a real record.

Treat AI import UX as governance. Default-deny uncertain rows. Validity rules follow the live section schema (visible columns), not a hard-coded spreadsheet layout. Footer status separates "selected" from "valid to import." Prompt schema and UI selection policy are one product surface.

Figure 1. Import UX is governance. Selection policy and field validity belong with the live schema, not a default-select-all. Source: Microsoft Learn: create a document processing model.

Figure 2. Who may import, and which rows count as valid, should be enforceable policy rather than a UI suggestion. Source: Microsoft Learn: create a document processing model.

A migration checklist that only compares row counts will greenlight wrong money, wrong dates, and wrong statuses. Row counts catch catastrophic drops. They miss the quiet CAST that turns decimals into floats, the null that became empty string, and the column that silently defaulted.

Push MLV cutovers to per-column tables and type-aware aggregates. Borrow the tiered reconciliation classical data migrations already trust: rows, then keys, then column aggregates. Automate it. Treat the artifacts as release evidence, not as a screenshot buried in chat.

Figure 1. Row counts catch catastrophes. Column aggregates and quality reports catch the quiet CAST that breaks money. Source: Microsoft Learn: MLV data quality reports.

Figure 2. Treat per-column reconciliation artifacts as release evidence for MLV cutovers, not optional notebook scratch. Source: Microsoft Learn: MLV data quality reports.

Large facts can "succeed" on every run and still lie. Row counts look fine. Pipelines are green. Measures drift because deletes, late arrivals, and corrections were never part of the incremental contract. Analysts discover it weekly. Engineers call it flaky. It is usually a missing agreement.

Treat refresh drift as a contract break. Align watermark semantics with business expectations for corrections. Fast materialization without drift tests just fails faster. Publish a refresh SLA per fact grain. Make drift checks part of the pipeline, not a weekly analyst surprise. Sales-line style facts are the classic patient; the disease is mismatched notebooks, watermarks, and semantic consumers.

Figure 1. Green refreshes are not the same as honest facts. Contracts must cover late arrivals and corrections. Source: Microsoft Learn: data refresh in Power BI.

Free-text export columns are where names, phones, and emails hide. The same columns often hold codes and addresses that policy still allows. Blanking everything is lazy. Sprinkling regexp_replace through a notebook is how two engineers redact differently on the same Friday.

Put the redaction contract in one config cell the rest of the notebook reuses. Flags, tokens, patterns, and known-name helpers become reviewable. Residual risk decisions leave chat history and enter the artifact you promote.

Figure 1. PII handling is a governance contract. Label scopes and redaction rules belong in reviewable config, not tribal notebook lore. Source: Microsoft Learn: sensitivity labels.

Figure 2. Keep redaction config next to the notebook that owns the export path so reviewers can see what ships with the artifact. Source: Microsoft Learn: how to use notebooks.