Databricks Data Lineage: From Unity Catalog to Your Entire Stack
Quick definition: What is Databricks data lineage?
Databricks data lineage is the record of how data moves through Databricks: where it originates, how it transforms, and where it goes next. Unity Catalog captures this automatically, down to the column level, for workloads that run on Databricks. Tracing the same data before it enters or after it leaves Databricks means extending lineage beyond Unity Catalog.
If you run Databricks, you almost certainly already have lineage. Unity Catalog (Databricks’ built-in governance and metadata layer) captures it the moment a query runs, and for data that lives and moves inside Databricks, it works well.
The harder question is what happens to that lineage at the edges of the platform, where data arrives from systems you do not control and leaves for tools Unity Catalog was never built to see. That edge is where cross-platform data lineage becomes the real problem to solve. The cost of leaving it unsolved is quiet but real: broken dashboards that take days to trace, schema changes shipped without knowing what they break, and audits that stall at the first system Unity Catalog cannot see.
How does Unity Catalog capture data lineage?
Unity Catalog captures lineage automatically at runtime. When a query runs on Databricks, it captures the tables and columns involved and builds the graph without anyone instrumenting code or standing up a separate service.
According to the Databricks documentation, lineage:
- Covers queries in any supported language
- Aggregates across every workspace attached to the metastore
- Appears in the Lineage tab of Catalog Explorer in near real time
To capture this lineage, tables must be registered in a Unity Catalog metastore and queried through the Spark or Databricks SQL interfaces.
The graph is not limited to tables. Unity Catalog also captures the notebooks, jobs, and dashboards tied to a query, and it exposes the same lineage through the lineage system tables, so teams can programmatically query lineage data rather than only clicking through Catalog Explorer.
Lineage inherits the Unity Catalog permission model, which means people see lineage only for assets they already have access to. Because capture is automatic, the graph stays current as queries run, without the stale-documentation problem that manual lineage mapping always drifts toward.
For the work that happens inside Databricks, this is genuinely strong. Column-level lineage makes three jobs far easier:
- Impact analysis before a schema change
- Root cause investigation when a number looks wrong
- Tracing sensitive data flows when an auditor asks.
For a team whose data estate is entirely Databricks, this is often enough on its own. Unity Catalog lineage, paired with its access controls and quality expectations, covers the ground that matters when every asset lives in one platform and never leaves it.
The cross-platform question only starts to bite when data routinely crosses the Databricks boundary. For most enterprises today, it does, and often in both directions at once.
Where Databricks lineage stops
Unity Catalog lineage is bounded by Databricks: Data shows up from production databases, third-party feeds, and upstream pipelines, and it moves on to Snowflake models, BI tools like Tableau and Power BI, and downstream feature stores. Unity Catalog sees the Databricks leg of that trip. It does not see what happened before the data landed, or what happens once it leaves.
When your data crosses the boundary in either direction, automatic capture stops. That fits what Unity Catalog was built for, which is governing the Databricks estate. But it gets in the way of root cause and impact analysis, because those questions follow the data wherever it goes. A wrong number on a dashboard often traces back to two systems upstream of Databricks, and a schema change in Databricks can break a report three systems downstream.
Say a revenue figure starts in a production Postgres database, lands in Databricks through an ingestion pipeline, gets reshaped into a curated table, then feeds a Snowflake mart and a Tableau dashboard. When the number looks off, Unity Catalog shows you the transformations inside Databricks. It will not tell you that the Postgres source changed a column definition last week, or that the Snowflake mart applies a different filter. Both sit outside Databricks, and that is where the explanation lives.
Does Unity Catalog track lineage across other platforms?
It can, in two limited ways, and they do different jobs. That difference is easy to miss, especially as Databricks positions Unity Catalog as a cross-platform layer rather than a Databricks-only catalog.
Lakehouse Federation
The first mechanism is Lakehouse Federation. It lets Databricks query external databases through read-only connections, without moving the data. That is useful, but it is a way to reach external data for compute, not a way to capture that data’s lineage.
Being able to query a table that lives in another system is not the same as knowing where it came from, how it was transformed, and what it breaks downstream. The two get conflated because both touch external systems, but federation answers “can I read this data?” while lineage answers “where did this data come from and what depends on it.”
External lineage (“bring your own data lineage”)
External lineage (which Databricks brands “bring your own data lineage”) lets you register external assets as metadata objects in Unity Catalog and define their relationships by hand, so they appear in the graph alongside Databricks assets. This is the closest Unity Catalog comes to cross-platform lineage. It is also manual: you create each external object, map its columns, and draw its relationships yourself.
In practice, a metastore admin grants the privilege to create external metadata, and someone defines each object and its relationships through Catalog Explorer, the API, or the SDK. That work then has to be maintained as schemas and systems change, because nothing updates those external definitions automatically.
So neither path is automated cross-platform lineage capture. One federates access for querying. The other depends on people registering and maintaining external metadata by hand.
The hidden cost of manual lineage maintenance
Registering external assets in Unity Catalog by hand is not a one-time project. Every schema change, every new pipeline, every additional source system means someone has to update the external metadata objects, remap the columns, and redraw the relationships, or the lineage graph drifts out of sync with production. That maintenance burden falls on the same engineering team already managing the data estate. In practice, it means lineage accuracy becomes a function of engineering bandwidth rather than a property of the system.
Teams that have tried to scale manual lineage maintenance consistently hit the same wall. The graph reflects last month’s architecture, not today’s, and the moment someone relies on it for a schema change or an incident investigation, the gap between documented and actual becomes the problem.
DataHub’s connectors capture lineage automatically as data moves, so the graph stays current without anyone owning the upkeep. The engineering time that was going into maintaining external metadata registrations goes back to the work that actually moves the data estate forward.
| Unity Catalog mechanism | What it does for lineage | How it works |
| Automatic runtime lineage | Captures table- and column-level lineage inside Databricks | Built automatically as queries run |
| Lakehouse Federation | Enables querying of external data | Read-only connections for compute, not lineage capture |
| External lineage (“bring your own lineage”) | Represents external assets in the graph | Manual registration and relationship mapping |
What complete cross-platform lineage requires
Lineage that works across your whole stack needs four things a single-platform tool cannot give you:
- Automated capture across every connected platform, so lineage does not depend on anyone hand-registering each asset or remembering to update it when something changes.
- Real-time propagation, so a change anywhere updates the graph in seconds rather than on a schedule or at query time, and impact analysis reflects production reality instead of last night’s snapshot.
- Bidirectional flow, so the system both ingests lineage from source systems and pushes business context back to them, keeping definitions consistent rather than duplicated across tools.
- Column-level precision that holds across platforms, so you can trace a single field from its source table through every transformation to the report or model that uses it, no matter how many systems it crosses.
This is the split that makes Unity Catalog and a dedicated lineage layer complementary rather than competing:
- Unity Catalog gives you governance depth inside Databricks
- A context platform gives you lineage and context breadth across everything else, Databricks included
Neither alone shows the full journey, and the teams already running both are proving the combination works.
The payoff shows up in operations. In an IDC study of DataHub Cloud (March 2026), organizations mapped lineage across 75% more of their datasets, resolved data-related outages 58% faster, and saw 56% fewer data completeness issues.
There is an AI dimension to this, too: Analytics agents (and those who rely on their answers) can only trust an output if the data behind it has traceable provenance. When lineage stops at the Databricks boundary, an agent reasoning over data that crossed that boundary is working without the full picture. Complete lineage is part of what lets you stand behind the answer.
Using DataHub and Unity Catalog together
DataHub does not replace Unity Catalog. It extends it, giving you one lineage graph that spans Databricks and everything connected to it.
Automatic ingestion
DataHub’s Unity Catalog connector ingests metadata and lineage from Databricks automatically, then continues the graph in both directions: upstream to the systems data came from, and downstream to the BI and ML tools it feeds.
Where Unity Catalog’s external lineage asks you to register and maintain outside assets by hand, DataHub’s connectors capture them automatically as data moves, so the cross-platform view does not depend on manual upkeep.
Real-time propagation
When someone documents a column in DataHub, a description or a governance label like a glossary term, that context propagates automatically down the lineage graph to the downstream and sibling columns derived from it, across whatever platforms those assets live on. You document a field once at its source, and the context follows it everywhere that field ends up, instead of being rewritten on each derived table.
Where Unity Catalog’s external lineage asks you to register and update outside assets by hand, propagation through DataHub’s lineage graph does that automatically, so governed definitions stay consistent as your data estate grows, without anyone chasing schema changes across platforms.
Bidirectional sync
Propagation carries context down the graph. Sync writes it back into the catalogs your teams already use. Tags and descriptions applied in DataHub sync back into Unity Catalog itself, on tables, columns, catalogs, and schemas, and they are removed from Unity Catalog when you remove them in DataHub. The same automation keeps other catalogs like Snowflake and BigQuery aligned.
So your team documents context once in DataHub instead of maintaining it separately in Unity Catalog, the BI tools downstream, and everywhere else the data lands. Databricks users still see governed descriptions and tags in Unity Catalog, with DataHub as the source of truth behind them.
Event-driven active metadata
DataHub keeps the lineage graph current through event-driven active metadata. Rather than waiting for a scheduled crawl, it updates lineage as data changes across connected systems, so it reflects production now instead of the last refresh. The result is a single column-level lineage graph that spans source systems, Databricks, and the tools downstream, filterable by owner, platform, or time.
Impact analysis
That single graph changes day-to-day work. Before a schema change, engineers can see the full blast radius, every downstream dashboard, model, and owner affected, even when those assets live in other platforms. Because the graph is column-level, that blast radius is precise to the individual field, not just the table.
Funding Circle gave more than 300 users self-service impact analysis across 23,000+ datasets, with column- and table-level lineage connecting producers to the consumers who depend on their data.
Discovery
Discovery runs on the same graph. An analyst can trace a dashboard upstream to the source-of-truth dataset feeding a metric, instead of guessing between three similarly named tables, and the lineage does the work that keyword search alone misses. Foursquare cut its time to data discovery from days to minutes across a stack that includes Databricks.
One lineage layer across Databricks and Snowflake
For teams running both Databricks and Snowflake, DataHub becomes the single place to trace lineage across the two. It shows what feeds a Snowflake model and what a schema change in Snowflake breaks downstream, with those assets sitting in the same graph as the Databricks ones.
With DataHub’s MCP server, a natural-language question about data routes to the right engine, Databricks Genie or Snowflake Cortex, based on where the data actually lives. Those agents are only as accurate as the lineage behind them, and because that lineage spans both platforms, they answer from the full path the data took rather than one system’s slice of it. A context layer like Databricks Genie Ontology grounds Genie in what the data means, the trusted definition of a metric, while lineage is what lets you trace where that number came from and what breaks when an upstream table changes. It also keeps tags and documentation in sync across both systems, so the governance policies built on that context stay consistent rather than diverging platform by platform.
Unity Catalog gives you depth inside Databricks. A cross-platform lineage layer gives you the rest of the picture, the part that starts before data reaches Databricks and continues after it leaves. Together they show the whole journey, which is what acting on lineage with confidence actually requires.
