24 posts tagged with "Microsoft Fabric"

Inconsistent workspace display-name prefixes look cosmetic until CI bindings, support scripts, and onboarding docs all grow translation layers. Every new engineer learns a dialect map. Every pipeline encodes yesterday's politics in a string.

Rename the estate once. Display names are for humans; catalog and lakehouse names may stay stable on purpose. Prefix chaos multiplies CI, docs, and scripts. Do the rename as a program with an old-to-new map. Strip legacy vendor codes when they no longer mean ownership. Naming standards belong in platform docs next to Variable Library maps. Perpetual translation is more expensive than a planned rename.

Figure 1. Workspace and lakehouse naming is estate architecture. Display names should guide humans without freezing bad prefixes forever. Source: Microsoft Learn: workspaces.

Re-CASTing the same physical column in every medallion layer feels careful. It is usually how drift multiplies. Bronze casts to decimal. Silver casts again "to be safe." Gold casts a third time and invents an unknown literal with yet another type. Consumers inherit a guessing game.

Type each published column once, as early as you dare (usually bronze, sometimes the first stable clean view). Silver and gold keep aliases, business calculations, and typed unknown-row literals only. UNION alignment casts are the exception, not the lifestyle. Lint the rest.

Figure 1. Type each column once, as early as you dare. Downstream layers should inherit the contract, not reinvent CASTs. Source: Microsoft Learn: copy data activity.

Copying tables into a lakehouse is the easy part. The hard part is whether 0 means zero or was null in the legacy system, and whether a 0.01 drift is rounding or a bug.

I treat migration as a semantic contract, not a file move. Stakeholders do not celebrate that Delta exists. They celebrate that last month's margin still matches within agreed rules.

Figure 1. Shortcuts and schemas change where data appears, not what it means. Source: Microsoft Learn: lakehouse schemas.

Current-state dimensions lie the moment a device moves. Dashboards that say "where is it now" are fine for operations. History questions ("where was it when the alarm fired?") need slowly changing dimension type 2, not a hope that someone kept yesterday's extract.

For measuring points and similar deployed assets, I put location and classification history in SCD2 Materialized Lake Views before anyone builds "devices as of" pages. The change feed is the source of truth. The MLV is the productized history contract.

Figure 1. Point-in-time placement is an SCD2 product question. Model effective dating before you ask dashboards to invent history. Source: Microsoft Learn: dimensional modeling dimension tables.

Figure 2. MLVs make history transforms operable. Lineage shows which change feed feeds the point-in-time contract. Source: Microsoft Learn: get started with materialized lake views.

A migration checklist that only compares row counts will greenlight wrong money, wrong dates, and wrong statuses. Row counts catch catastrophic drops. They miss the quiet CAST that turns decimals into floats, the null that became empty string, and the column that silently defaulted.

Push MLV cutovers to per-column tables and type-aware aggregates. Borrow the tiered reconciliation classical data migrations already trust: rows, then keys, then column aggregates. Automate it. Treat the artifacts as release evidence, not as a screenshot buried in chat.

Figure 1. Row counts catch catastrophes. Column aggregates and quality reports catch the quiet CAST that breaks money. Source: Microsoft Learn: MLV data quality reports.

Figure 2. Treat per-column reconciliation artifacts as release evidence for MLV cutovers, not optional notebook scratch. Source: Microsoft Learn: MLV data quality reports.

Your pipeline runs. You did not touch half the notebooks. They still publish.

That is not always a bug in your YAML. Fabric deployment models and item pairing often optimize for workspace consistency, not git-style minimal diffs. If you design as if only changed files move, production will teach you otherwise.

Figure 1. Deployment is a workspace operation. Source: Microsoft Learn: deployment pipelines.

Materialized Lake View runs paste warning floods that mention native execution engine fallbacks. Teams panic-rewrite Spark SQL to chase every line. Meanwhile the DAG's wall-clock still hides in one slow node nobody measured.

Measure first. Pasted warnings include Delta and MLV metadata internals you cannot and need not remove. Wall-clock attribution beats fear-driven rewrites. Fix the few fallbacks that move latency. Ignore the rest deliberately. Document accepted warnings so on-call does not thrash. Pair with per-view notebooks so slow nodes are isolatable.

Figure 1. Attribute MLV time with job details and DAG reality. Do not treat every NEE warning as a rewrite mandate. Source: Microsoft Learn: Spark detail monitoring.

Fresh gold tables are useless if the semantic model still points at yesterday, or if refresh burns an hour on a fact you could have incrementalized. Refresh is an architecture problem: orchestration, identity, and storage mode. Treat it as a portal checkbox and you invent a second process nobody can explain.

Figure 1. Direct Lake connects the semantic model to Delta in OneLake. Source: Microsoft Learn: Direct Lake overview.

Medallion architecture asks where bronze, silver, and gold live. In Fabric you can answer with many lakehouses or with schemas inside one lakehouse.

That choice leaks into shortcuts, SQL endpoints, notebooks, CI/CD, and how painful renames become later. It is not a folder preference. It is a topology decision.

Figure 1. Lakehouse creation in Microsoft Fabric. Source: Microsoft Learn: lakehouse schemas.

Fabric estates accumulate half-scripts: a notebook that checks capacities, a shell history that lists failed jobs, a wiki page with screenshots from last quarter. Each produces a different health story. On-call picks a favorite. Coding agents inherit the contradiction.

Consolidate into one operable skill. Clear verbs beat tribal shell history. Agents consuming monitor tools need stable output contracts. Separate diagnosis from remediation. Version runbooks like product code. Many half-scripts create contradictory signals; one interface creates a shared truth you can improve.

Figure 1. Monitoring should be an operable interface with stable outputs, not a pile of one-off scripts. Source: Microsoft Learn: monitoring hub.

Moving a handful of warehouse views to Materialized Lake Views is a rewrite. Moving hundreds is a program. Teams that treat the long tail like a weekend notebook usually discover dependency order the hard way: gold breaks, silver thrashes, and bronze shortcuts were never registered as the real contract.

Frame the work as bronze to silver to gold with gates. Count the estate. Order by dependency. Prove a representative POC. Then convert with per-view notebooks so blast radius stays local. Success is parity plus runtime, not "the portal shows an MLV."

Figure 1. MLV migration is a program across bronze, silver, and gold. Gates belong between layers, not as a weekend cutover wish. Source: Microsoft Learn: medallion lakehouse architecture.

InsertAll of multi-million ledger rows on one driver is a known failure mode. The notebook looks simple. The driver dies. Someone asks for a bigger cluster. The next reload dies later, with a larger bill.

Treat bulk reload as orchestration: batched parallel page ingest, progress monitoring, and entity filters for operable reruns. Separate one-time rebuild paths from incremental watermarks. Publish the pattern to prod only after batch-level observability exists. OOM here is a design smell in notebook API usage, not proof that "Fabric is flaky."

Figure 1. Multi-million InsertAll on one driver is an orchestration smell. Batch ingest belongs in an operable pipeline. Source: Microsoft Learn: copy data activity.

Figure 2. Batch with progress and entity filters, then promote the pattern like any other scheduled pipeline. Source: Microsoft Learn: pipeline overview.

Large facts can "succeed" on every run and still lie. Row counts look fine. Pipelines are green. Measures drift because deletes, late arrivals, and corrections were never part of the incremental contract. Analysts discover it weekly. Engineers call it flaky. It is usually a missing agreement.

Treat refresh drift as a contract break. Align watermark semantics with business expectations for corrections. Fast materialization without drift tests just fails faster. Publish a refresh SLA per fact grain. Make drift checks part of the pipeline, not a weekly analyst surprise. Sales-line style facts are the classic patient; the disease is mismatched notebooks, watermarks, and semantic consumers.

Figure 1. Green refreshes are not the same as honest facts. Contracts must cover late arrivals and corrections. Source: Microsoft Learn: data refresh in Power BI.

Free-text export columns are where names, phones, and emails hide. The same columns often hold codes and addresses that policy still allows. Blanking everything is lazy. Sprinkling regexp_replace through a notebook is how two engineers redact differently on the same Friday.

Put the redaction contract in one config cell the rest of the notebook reuses. Flags, tokens, patterns, and known-name helpers become reviewable. Residual risk decisions leave chat history and enter the artifact you promote.

Figure 1. PII handling is a governance contract. Label scopes and redaction rules belong in reviewable config, not tribal notebook lore. Source: Microsoft Learn: sensitivity labels.

Figure 2. Keep redaction config next to the notebook that owns the export path so reviewers can see what ships with the artifact. Source: Microsoft Learn: how to use notebooks.

Checklist reporting loves wide pivots. Teams often build them late: SQL OUTER APPLY, Import mode, captions invented in the report. Refresh time climbs. Direct Lake never gets a fair chance. The semantic model pays for a shape that should have been decided in gold.

Pivot once in gold. Broadcast joins plus a single groupBy/pivot in PySpark produce a self-contained fact: one row per reading with stable keys. Captions match the language users already see in PDFs. Materialize for Direct Lake. Stop paying Import tax for wide pivots that only exist because the warehouse view stayed nested.

Figure 1. Direct Lake wants report-ready flat facts, not nested checklist trees that only Import mode can fake. Source: Microsoft Learn: understand relationships.

Medallion diagrams still tempt teams to land a physical bronze copy of everything upstream. When the upstream domain already publishes curated Delta in OneLake, that copy is not prudence. It is a second warehouse with a friendlier name.

Bronze can be a contract of shortcuts and thin clean views. Ownership stays with the publisher. Freshness follows the source. Your CI/CD list becomes the shortcut set, not a shadow catalog of cloned tables.

Figure 1. Bronze can be a pointer surface. Prefer OneLake shortcuts over copying bytes the platform already holds. Source: Microsoft Learn: OneLake shortcuts.

Telemetry tables named after pipelines invite people to treat events like disposable logs. Reports then reinvent grain, matching, and captions in DAX. Volume pushes Import mode until refresh becomes the weekly outage.

Name the business concept (events). Enrich with the same matching logic used by service-device facts. Land them in gold for Direct Lake. Captions, query folders, and descriptions are part of the semantic handoff. Watermark shortcuts and clean views are part of the bronze-to-gold story, not footnotes.

Figure 1. IoT events are facts. Land them as modeled destinations, not as log piles that only Import mode can fake into a report. Source: Microsoft Learn: Eventstreams overview.

Nine warehouse views timing out usually means two trunks and seven dependents. Teams that rewrite every leaf, or jump straight to a new platform, often miss the cheap win: materialize the consolidations everything else hangs from.

Use a layered playbook. Drop orphans. Apply cheap predicate and join rewrites. Then materialize the two or three heavy consolidations. Measure with the same harness before and after. Document what is a view versus a managed table so CI/CD stays honest.

Figure 1. When dependents time out, materialize the trunk consolidations first instead of rewriting every leaf view. Source: Microsoft Learn: data warehousing in Fabric.

Figure 2. Measure with the same harness before and after. Document view versus managed table so CI/CD stays honest. Source: Microsoft Learn: data warehousing in Fabric.

Hard-coded workspace IDs in notebooks are how Test accidentally rebinds Prod. Spreadsheet "environment maps" in chat are how two teams maintain different truths. Fabric already ships a better home: a Git-tracked Variable Library with per-stage value sets.

Put Test, Preprod, and Prod values in the item. Teach rebind notebooks to read via supported APIs. Split platform versus domain workspaces explicitly. Respect Git schema so restores work. Pair the library with a promotion story when portal parameters regress.

Figure 1. Promotion is safer when workspace targets come from a Variable Library map, not pasted GUIDs. Source: Microsoft Learn: Variable Library overview.

Parameterized DataLake expressions can work for months and then fail across stages overnight. Teams panic-rewrite convention docs as if literal OneLake URLs were always the design goal. That erases history and guarantees the workaround becomes religion.

Reframe "literal URL required" failures as service regressions. Ship validators and literal patches as temporary controls. Keep a Test to Preprod to Prod promotion story under the workaround. File and track the Microsoft support trail next to the repo convention. Do not pretend the platform always wanted hard-coded URLs.

Figure 1. Direct Lake on OneLake is the intended pattern. Literal URL patches are controls when parameters regress. Source: Microsoft Learn: Direct Lake storage.

Helpful gold views that "just join service and device" hide many-to-many grain until a report page disagrees with itself. Filters double count. Role-playing dims break. Two pages that should reconcile never do.

Deploy bridge keys as a first-class artifact and remodel the service-device fact before you polish visuals. Matching logic must be shared with telemetry enrichment or the numbers diverge by construction. Semantic role-playing dims need matching foreign key types. Remodel first. Visual polish second.

Figure 1. Many-to-many grain leaks into blank or double-counted visuals. Remodel the fact before polishing pages. Source: Microsoft Learn: understand relationships.

Figure 2. Bridge keys and filter direction are part of the remodel. Dashboards inherit whatever grain you leave in gold. Source: Microsoft Learn: understand relationships.

Pipeline graphs that live in people's heads do not survive vacation handoffs. Ad-hoc docs rot. A "harmless" bronze rename still nukes gold because nobody saw the edge.

Build a small lineage explorer over metadata you already emit: notebooks, MLVs, shortcuts, pipelines. Read-only exploration is enough for v1. Treat lineage UX as platform product, not a slide in an architecture deck.

Figure 1. Lineage that lives in people's heads does not survive a rename. Explore metadata before you change identities. Source: Microsoft Learn: lineage in Fabric.

Figure 2. A thin read-only explorer over notebooks, MLVs, shortcuts, and pipelines is platform product, not a slide. Source: Microsoft Learn: lineage in Fabric.

A gold device table that looks clean in Spark can still fail as a platform asset. Optional links to contracts drop unmatched devices. Free-text site fields hide business keys behind tribal regex. Captions and folders are treated as polish. Stakeholders open a report and half the estate is missing without an error.

Gold becomes useful when unmatched keys, captions, and relationships ship as one semantic contract with the table. "All devices visible regardless of match" is an acceptance test, not a slogan.

Figure 1. Gold only becomes a platform asset when keys and relationships behave as a semantic contract the model can trust. Source: Microsoft Learn: understand relationships.

Figure 2. Row counts on the SQL endpoint can look healthy while reports hide unmatched keys. Test the contract, not only Spark success. Source: Microsoft Learn: lakehouse SQL analytics endpoint.

Fabric notebooks and pipelines are great at doing work. They are mediocre at being the calendar, the dependency graph, and the on-call story for everything around them. The moment three jobs must finish before a fourth can start, you need an orchestration plane that is not pretending to be another transform engine.

Airflow fits when you want an honest DAG: schedule, dependencies, retries, and a run history humans can read. Keep tasks thin. Call Fabric. Do not rebuild Fabric inside operators because it was convenient on a Friday.

Figure 1. Fabric Airflow jobs can trigger notebooks, pipelines, and other items from a DAG without stuffing transform logic into the orchestrator. Source: Microsoft Learn: run a Fabric item using Apache Airflow.