One column-level lineage graph across production systems, dbt, Airflow, Snowflake, Databricks, Redshift, BI tools such as Tableau, Power BI, and Looker. Understand all the implications of deprecating a legacy table.
"What reads this table?" used to be answered by asking around, and that held up while nothing moved. Here is what four teams said the week a cutover date landed on them.
We're moving to Snowflake and we have no idea what depends on what.
Scope becomes a guess Timelines slip on discovery, not on engineering.
We can't decommission the legacy warehouse because nobody can prove what still reads from it.
Dual-running never ends You keep paying for both platforms, indefinitely.
We have three catalogs and no owner.
Nothing is authoritative Three tools drift and none of them is trusted.
dbt gives us lineage inside dbt. It stops at the boundary.
The risk sits in the handoff Which is where migrations break things.
Recognize any of these? We'll walk the same situation through DataHub.
Book a demoAll seven hops, in one column-level graph. Change the Postgres column and DataHub names the Tableau dashboard that will be wrong on Monday, plus the dbt model, the Databricks table and the Snowflake mart in between.
Ingest the source and target platforms and let the SQL parser read the query logs. You get the real dependency map, including the things nobody remembered, before you commit to a date. You also get the list of assets nobody has touched in months and that don't need migrating at all.
Column-level impact analysis runs upstream or downstream and filters by platform, entity type and owner. Before you repoint a model you know which dashboards, feature tables and downstream jobs it touches, and who to tell. Descriptions and governance labels propagate to the new assets instead of being retyped.
Usage and query history show which legacy tables still have live consumers and which have had none for months. You retire the old platform against evidence you can put in front of a steering committee, and stop paying to run two stacks in parallel.
Unity Catalog, Snowflake Horizon and dbt's own docs are good at what they do. None of them is trying to be the catalog for the systems it doesn't own, which is why you ended up with three. Pick a tool to see what it covers, where its picture stops, and what a cross-system layer adds.
It manages the native Snowflake metadata surface: objects, tags, policies and access. That should not change.
Native context capabilities operate inside the warehouse. Most of a migrating estate does not: the ingestion that feeds it, the transformation engine, the BI layer and the platform you're retiring all sit outside.
Bidirectional metadata sync means the two work together rather than competing. DataHub ingests dbt models with column-level lineage propagating through the transformation logic, and accepts External Lineage API events so lineage in Snowflake reflects upstream work.
Lineage is captured automatically for workloads that run on Databricks, and it captures the Databricks leg accurately.
Data that originates outside Databricks or leaves for other systems falls outside that automatic capture. Lakehouse Federation lets you query external data but is not a way to capture its lineage, and external lineage has to be registered and mapped by hand.
One column-level lineage graph spanning source systems, Databricks and the tools downstream, captured automatically as data moves, so the cross-platform view doesn't depend on manual upkeep. Tags and descriptions applied in DataHub sync back into Unity Catalog on tables, columns, catalogs and schemas.
The DAG, the tests and the model docs belong in dbt. DataHub reads them rather than asking you to duplicate them.
dbt's graph knows what dbt built. It doesn't know the ingestion job that filled the source table, the second dbt project another team runs, or the dashboards and reverse-ETL syncs reading the output. That is exactly the surface a migration disturbs.
Run ingestion for both dbt and your warehouse and DataHub automatically creates sibling relationships between the dbt nodes and the warehouse nodes. They appear as a single entry carrying both platform logos, with column-level lineage extracted from the parsed SQL.
Standardising on Airflow is usually part of the same consolidation program. DataHub attaches to it as a plugin rather than asking it to change.
pip install acryl-datahub-airflow-pluginAirflow can tell you a DAG succeeded. On its own it can't tell you which columns that task rewrote, what reads them, or whether the run you're about to migrate is the one holding up a legacy table's retirement.
The plugin extracts column-level lineage automatically from supported SQL operators, including Postgres, MySQL, Snowflake, BigQuery and Redshift, and resolves temporary-table dependencies when multi-statement SQL parsing is on. Task run results land alongside the assets they produced.
Consolidation here means one place to look, not one vendor to buy everything from. Can you still use Horizon if you use DataHub? Yes. Horizon manages the native Snowflake metadata surface, and the two are designed to work together. The same holds for Unity Catalog and for dbt. What you remove is the third and fourth general-purpose catalog nobody owns, and the spreadsheet that reconciles them.
Ask where the boundary sits for a stack like yours. We'll show you in the product.
Book a demo-- 1. a role for DataHub to read metadata with create role if not exists datahub_role; grant operate, usage on warehouse "COMPUTE_WH" to role datahub_role; -- 2. the service user it connects as create user datahub_user display_name = 'DataHub' type = LEGACY_SERVICE default_role = datahub_role; -- 3. read-only metadata grants, per database grant usage on database "<database>" to role datahub_role; grant usage on all schemas in database "<database>" to role datahub_role; grant select on all tables in database "<database>" to role datahub_role; grant references on all views in database "<database>" to role datahub_role; grant imported privileges on database snowflake to role datahub_role;
Three statements: a role, a service user, and read-only grants per database. Nothing writes back to the warehouse.
Configure the recurring ingestion pipeline in the UI or in YAML, then schedule it. It pulls databases, schemas, tables, external tables, views and materialized views.
Table and column lineage, usage statistics from recent query activity, and table profile statistics come from the same connection. Where a platform has no native tracking, the SQL parser reads the query log instead.
dbt, Airflow, Databricks, Redshift, Looker, Tableau and Power BI each connect the same way. The graph joins them automatically: one estate, added a platform at a time.
Every system on both sides of the cutover, plus the tools reading from them.
Counts from the DataHub connector directory, which also covers data lakes, messaging, data quality, AI/ML and identity.
Want to see the whole thing working end to end?
Book a demoDPG Media grew by acquisition, and every acquisition brought new systems and overlapping data into the group. The team had been operating much like a data mesh already, but without any form of governance on top of it. Once DataHub Cloud made usage and duplication visible across Snowflake, Redshift, Looker, Tableau, Qlik, Kafka, Spark, Airflow and dbt, they could identify and eliminate unused and duplicate data, and take the warehouse bill down with it.
"With DataHub, we were able to reduce our Snowflake costs by 25% each month."
A real consolidation estate, not a single-warehouse demo.
Sources: DPG Media customer story and the DataHub Snowflake partnership page.
Share a few details about your goals, and we'll be in touch to schedule a personalized walkthrough tailored to your data environment, use cases, and roadmap.
We'll cover:
Let's get you the answers and insights you need.
A DataHub specialist will be in touch to schedule your walkthrough. Takes 30 minutes, no prep required.
Prefer to look around first?
Take the self-guided product tour
No data required. Demos run in a DataHub environment on a realistic estate. Nothing connects to your systems, and we don't work from data you provide.