Lineage explorers beat tribal notebooks
Pipeline graphs that live in people's heads do not survive vacation handoffs. Ad-hoc docs rot. A "harmless" bronze rename still nukes gold because nobody saw the edge.
Build a small lineage explorer over metadata you already emit: notebooks, MLVs, shortcuts, pipelines. Read-only exploration is enough for v1. Treat lineage UX as platform product, not a slide in an architecture deck.
Figure 1. Lineage that lives in people's heads does not survive a rename. Explore metadata before you change identities. Source: Microsoft Learn: lineage in Fabric.
Figure 2. A thin read-only explorer over notebooks, MLVs, shortcuts, and pipelines is platform product, not a slide. Source: Microsoft Learn: lineage in Fabric.
I learned this watching careful engineers break dependents they did not know existed. The Fabric portal lineage views helped for some items and still left gaps across notebooks and custom jobs. A thin explorer over our own metadata paid for itself the first time someone checked before renaming.
The core idea
If engineers cannot see dependencies before they change an object, you are running an oral tradition with Spark bills.
Start from metadata you already produce. Do not wait for a perfect graph database. Read-only search and neighborhood views prevent the worst own-goals. Editing lineage can wait. Platform teams should version and ship the explorer like any other internal product: owners, backlog, feedback.
A model that stays explainable
1. Inventory edges you already have
Notebooks declare lakehouse defaults and table writes. MLVs declare sources. Shortcuts declare targets. Pipelines declare activities. Parse what you can into a simple edge list. Incomplete is fine. Invisible is not.
2. Ship read-only neighborhood views first
Given an object, show upstream and downstream one or two hops with links to repo paths or workspace items. Search by name. That alone stops many blind renames. Fancy graph physics can come later if ever.
3. Prefer boring storage
A table or index rebuilt on a schedule beats a fragile real-time fantasy for v1. Freshness measured in hours is often enough for engineering hygiene. Document freshness so people do not treat the explorer as instantaneous truth during an incident.
4. Keep humans in the rename loop
Explorer plus checklist: before rename/drop, open dependents, ping owners, update contracts. The tool does not replace change management. It makes change management possible at estate scale.
5. Leave editing for later
Write-back to rewrite pipelines from a GUI is a different product. V1 answers "what breaks if I touch this?" Scope creep into design-time editing kills the experiment.
6. Productize ownership
Name an owner, a channel for bugs, and a release cadence. Orphan internal tools become another tribal notebook. Lineage UX deserves the same seriousness as a shared ingest library.
Failure modes I design against
Docs as graph. Markdown lists that diverge weekly.
Portal-only assumption. Gaps across notebooks and custom jobs ignored.
Perfect schema before any UI. Months of modeling, zero prevention.
Write-back v1. Ambition collapses the experiment.
No owner. Explorer rots; trust dies; people stop looking.
Treating explorer as authorization. It shows edges; it does not grant access.
A thin vertical that proves value
Pick one domain lakehouse and the gold tables that feed two semantic models. Ingest edges for that slice. Give five engineers the search box during a refactor week. Measure whether they find dependents they would have missed. If yes, widen. If no, fix metadata coverage before expanding UI chrome. Experiments should answer a usage question, not only an architecture question.
Metadata quality is the real backlog
Garbage object names and missing notebook declarations produce garbage graphs. Pair the explorer with lightweight standards: required lakehouse binding comments, MLV source lists in repo, shortcut inventory files. The explorer makes missing metadata visible; standards reduce missingness. Without that pairing, people blame the tool for telling an honest incomplete story.
What to show on the object page
For each node, show type (notebook, MLV, shortcut, pipeline, table), owners if known, last graph build time, and links out. For each edge, show how it was inferred (parsed write, declared source, pipeline activity). Inferred edges should look different from declared edges so engineers know the confidence. That small honesty does more for trust than a prettier layout.
Rollout without boiling the ocean
Announce the explorer as best-effort coverage. Invite teams to add metadata where gaps hurt them. Celebrate the first caught rename in the platform channel. Culture changes when the tool prevents a pain people remember, not when a wiki page declares lineage important.
Security and tenancy boundaries
The explorer should respect the same workspace access the engineer already has. Do not build a god-mode metadata dump that bypasses Fabric permissions. If you aggregate across workspaces, filter to what the caller can see. A lineage tool that leaks names across isolation boundaries will get shut down, and it should.
Trade-offs
Building even a thin explorer costs engineering time. Not building it costs outages and fear-driven freezes ("nobody touch bronze"). Scheduled rebuilds lag reality; live parsing is harder. Start scheduled. Incomplete graphs can mislead if freshness and coverage are not labeled; label them prominently. Read-only frustrates people who want edits; protect v1 scope anyway.
What I would put on an ADR
- Platform ships a read-only lineage explorer over notebook/MLV/shortcut/pipeline metadata.
- V1 scope is search plus upstream/downstream neighborhood views.
- Graph freshness and coverage limits are displayed in the UI.
- Rename/drop runbooks require an explorer check for critical objects.
- Editing/write-back is explicitly out of scope for v1.
- Named owner and release cadence for the explorer as an internal product.
Primary references: Fabric lineage and monitoring hub concepts on Microsoft Learn. Pair portal views with your estate-specific metadata so gaps across notebooks are not wished away.
Closing
Tribal notebooks do not scale past the people who wrote them.
Give engineers a way to see edges before they cut them. Start read-only, start incomplete, start owned. Lineage stops being a slide and becomes a habit that saves gold from "harmless" bronze renames.