Fabric PII: What Would You Do When Ingesting Sensitive Data?

Free-text export columns are where names, phones, and emails hide. The same columns often hold codes and addresses that policy still allows. Blanking everything is lazy. Sprinkling regexp_replace through a notebook is how two engineers redact differently on the same Friday.

Put the redaction contract in one config cell the rest of the notebook reuses. Flags, tokens, patterns, and known-name helpers become reviewable. Residual risk decisions leave chat history and enter the artifact you promote.

Figure 1. Redaction as a publish gate. Shared config, stable tokens, and a write path that cannot run when scrubbing was skipped.

Context and constraint

A common failure: export notebooks where one run scrubs phones and another leaves them in a comment field. The lakehouse is fine. The publish path is folklore. Privacy reviews ask what we redact. The honest answer is "whatever the last person remembered."

Anything that leaves the lakehouse boundary should pass through the same rules. Config-driven redaction makes those rules visible: what you strip, what you keep, which token replaces what, and which residual risks you accepted on purpose.

Figure 2 is the Learn surface for sensitivity label scopes. Labels classify and control access. They do not replace a notebook rewrite on the export path.

Figure 2. PII handling is a governance contract. Label scopes and redaction rules belong in reviewable config, not tribal notebook lore. Source: Microsoft Learn: sensitivity labels.

Figure 3 is why the redaction config belongs next to the notebook that owns the export. Reviewers need to see what ships with the artifact.

Figure 3. Keep redaction config next to the notebook that owns the export path so reviewers can see what ships with the artifact. Source: Microsoft Learn: how to use notebooks.

Design

1. One config cell, many consumers

Declare enable flags, column lists, regex patterns, replacement tokens, and optional UDF hooks once near the top. Later cells import that config. Do not re-implement phone scrubbing three times with three different patterns. Drift between cells is a privacy bug dressed as style. If a second notebook exports the same grain, share a module or notebook snippet so the contract is not copy-pasted into entropy.

2. Separate policy-allowed from policy-forbidden in free text

Names, personal phones, and personal emails usually go. Reference codes, operational identifiers, and many postal addresses may stay under a written policy. Encode that split with examples. A single "scrub all strings" hammer destroys analytical value and trains people to bypass the notebook. Bypass then becomes the real production path.

Figure 4 is the split you write once so every extract stops rediscovering it.

Figure 4. Free-text policy split. Argue personal versus operational once, write examples, and stop rediscovering the rule per extract.

3. Stable, reviewable tokens

Replace matches with tokens like {{EMAIL}} or {{PHONE}} that downstream can count and audit. Random deletion makes gap detection hard. Stable tokens also help support prove an extract was processed when someone asks whether a name could have survived. Count tokens in tests the way you count rows in parity jobs.

4. Known-name UDF as a deliberate layer

Regex will miss informal names and catch false positives. A known-name list or UDF is a second pass with ownership: who updates the list, how often, and what happens on misses. Document residual risk for titles and honorifics if you leave them alone on purpose. That decision belongs in the contract, not in a Slack thread from last quarter.

5. Treat redaction as a gate, not a nicety

Wire the notebook so publish fails closed when the redaction cell is skipped or flags are off in prod. Optional scrubbing in debug mode is fine locally. Optional scrubbing in the path that writes to a share location is a control failure. Make the output path depend on a successful redaction step in the same run.

6. Test with fixtures that include near-misses

Keep a small fixture set: obvious emails, phone formats you care about, codes that look like phones, addresses that must survive, and names adjacent to titles. Assert token counts and surviving allowed strings. Screenshot QA is not a privacy control. Re-run fixtures when someone "improves" a pattern.

Figure 5 is the four-part operating loop that keeps the gate honest after the first merge.

Figure 5. Stable tokens, an owned known-name pass, fixture near-misses, and a publish gate that fails closed. Metrics belong in the run output reviewers skim.

Publish gate mechanics that hold up

Wire the export path so the write to the outbound location cannot run unless redaction completed with flags appropriate for that stage. Local debug may skip with an obvious banner. Shared workspace runs that produce partner files must not. Pair the gate with a simple metrics cell: rows processed, tokens emitted per type, columns touched. Those metrics belong in the run output reviewers skim before they approve a share.

Common mistakes

Cell-local regex archaeology. Nobody can answer what the notebook redacts without reading every cell.

Blanking addresses and codes that policy allows. Users demand a bypass. Bypass becomes the real path.

Unstable replacements. Sometimes delete, sometimes mask, sometimes hash. Auditors cannot reconcile.

Leaving titles and free-text notes undecided. "We will handle it later" means production exports ship with people names next to job titles.

Redacting only the pretty columns. The comment, description, and notes fields are where PII actually lives.

Trusting labels without a rewrite step. Labeled data still leaves if the export notebook never scrubbed it.

Use as-is or adapt

Use as-is

• All lakehouse-leaving extracts use a shared redaction config module or notebook cell contract.
• Allowed versus forbidden free-text categories are written with examples.
• Replacement tokens are stable and counted in tests.
• Residual-risk decisions (for example titles) are explicit in the contract.
• Publish paths fail if redaction is disabled outside approved debug modes.
• Fixture-based tests cover phones, emails, names, and policy-allowed codes.

Adapt before you use

A shared config is slower to change than a one-off replace. That friction is useful. Privacy rules should not be casual. False positives on codes require careful patterns and allowlists; expect iteration with domain owners. Known-name lists need operational ownership or they rot. Residual risk will remain. The goal is documented, bounded risk, not magical certainty. You will also spend time arguing whether a field is personal or operational. Have that argument once, write it down, and stop rediscovering it per extract.

Primary references: Microsoft Learn on data protection and governance in Fabric and sensitivity labeling. Pair platform labels with notebook gates; neither replaces the other.

Put the contract in the notebook config. Reuse it. Test it. Gate publish on it. Free-text exports stop being a weekly surprise.