PII redaction belongs in the notebook config, not in tribal knowledge
Free-text export columns are where names, phones, and emails hide. The same columns often hold codes and addresses that policy still allows. Blanking everything is lazy. Sprinkling regexp_replace through a notebook is how two engineers redact differently on the same Friday.
Put the redaction contract in one config cell the rest of the notebook reuses. Flags, tokens, patterns, and known-name helpers become reviewable. Residual risk decisions leave chat history and enter the artifact you promote.
Figure 1. PII handling is a governance contract. Label scopes and redaction rules belong in reviewable config, not tribal notebook lore. Source: Microsoft Learn: sensitivity labels.
Figure 2. Keep redaction config next to the notebook that owns the export path so reviewers can see what ships with the artifact. Source: Microsoft Learn: how to use notebooks.
I learned this shipping export notebooks where one run scrubbed phones and another left them in a comment field. The lakehouse was fine. The publish path was folklore. Privacy reviews asked what we redact. The honest answer was "whatever the last person remembered."
The core idea
Redaction is a publish gate with an explicit contract, not a courtesy regex in the last cell.
Anything that leaves the lakehouse boundary (CSV dump, partner share, ticket attachment, "just this extract") should pass through the same rules. Config-driven redaction makes those rules visible: what you strip, what you keep, which token replaces what, and which residual risks you accepted on purpose. Sensitivity labels help discovery. They do not rewrite the string you just emailed.
A model that stays explainable
1. One config cell, many consumers
Declare enable flags, column lists, regex patterns, replacement tokens, and optional UDF hooks once near the top. Later cells import that config. Do not re-implement phone scrubbing three times with three different patterns. Drift between cells is a privacy bug dressed as style. If a second notebook exports the same grain, share a module or notebook snippet so the contract is not copy-pasted into entropy.
2. Separate policy-allowed from policy-forbidden in free text
Names, personal phones, and personal emails usually go. Site codes, equipment identifiers, and many postal addresses may stay under a written policy. Encode that split with examples. A single "scrub all strings" hammer destroys analytical value and trains people to bypass the notebook. Bypass then becomes the real production path.
3. Stable, reviewable tokens
Replace matches with tokens like {{EMAIL}} or {{PHONE}} that downstream can count and audit. Random deletion makes gap detection hard. Stable tokens also help support prove an extract was processed when someone asks whether a name could have survived. Count tokens in tests the way you count rows in parity jobs.
4. Known-name UDF as a deliberate layer
Regex will miss informal names and catch false positives. A known-name list or UDF is a second pass with ownership: who updates the list, how often, and what happens on misses. Document residual risk for titles and honorifics if you leave them alone on purpose. That decision belongs in the contract, not in a Slack thread from last quarter.
5. Treat redaction as a gate, not a nicety
Wire the notebook so publish fails closed when the redaction cell is skipped or flags are off in prod. Optional scrubbing in debug mode is fine locally. Optional scrubbing in the path that writes to a share location is a control failure. Make the output path depend on a successful redaction step in the same run.
6. Test with fixtures that include near-misses
Keep a small fixture set: obvious emails, phone formats you care about, codes that look like phones, addresses that must survive, and names adjacent to titles. Assert token counts and surviving allowed strings. Screenshot QA is not a privacy control. Re-run fixtures when someone "improves" a pattern.
Failure modes I design against
Cell-local regex archaeology. Nobody can answer what the notebook redacts without reading every cell.
Blanking addresses and codes that policy allows. Users demand a bypass. Bypass becomes the real path.
Unstable replacements. Sometimes delete, sometimes mask, sometimes hash. Auditors cannot reconcile.
Leaving titles and free-text notes undecided. "We will handle it later" means production exports ship with people names next to job titles.
Redacting only the pretty columns. The comment, description, and notes fields are where PII actually lives.
Trusting labels without a rewrite step. Labeled data still leaves if the export notebook never scrubbed it.
Publish gate mechanics that hold up
Wire the export path so the write to the outbound location cannot run unless redaction completed with flags appropriate for that stage. Local debug may skip with an obvious banner. Shared workspace runs that produce partner files must not. Pair the gate with a simple metrics cell: rows processed, tokens emitted per type, columns touched. Those metrics belong in the run output reviewers skim before they approve a share.
Trade-offs
A shared config is slower to change than a one-off replace. That friction is useful. Privacy rules should not be casual. False positives on codes require careful patterns and allowlists; expect iteration with domain owners. Known-name lists need operational ownership or they rot. Residual risk will remain. The goal is documented, bounded risk, not magical certainty. You will also spend time arguing whether a field is personal or operational. Have that argument once, write it down, and stop rediscovering it per extract.
What I would put on an ADR
- All lakehouse-leaving extracts use a shared redaction config module or notebook cell contract.
- Allowed versus forbidden free-text categories are written with examples.
- Replacement tokens are stable and counted in tests.
- Residual-risk decisions (for example titles) are explicit in the contract.
- Publish paths fail if redaction is disabled outside approved debug modes.
- Fixture-based tests cover phones, emails, names, and policy-allowed codes.
Primary references: Microsoft Learn on data protection and governance in Fabric and sensitivity labeling concepts. Pair platform labels with notebook gates; neither replaces the other.
Closing
If redaction lives in tribal knowledge, you do not have a privacy control. You have a habit that disappears on vacation.
Put the contract in the notebook config. Reuse it. Test it. Gate publish on it. Free-text exports stop being a weekly surprise.