Data Lineage in ETL: What It Takes to Be Useful in Production

By: Lakshay Nasa

07.10.26

Data lineage in ETL is the record of how a piece of data moves and transforms across the extract, transform, and load stages of a pipeline. The structural pattern is consistent: source datasets are consumed by ETL jobs, those jobs transform the data, and the resulting data is written to target datasets, producing a dependency graph that links every field to its origin.

In practice, that graph answers three questions:

Where did this field originate?
What happened to it on the way here?
Where is it consumed downstream?

That is a narrower scope than data lineage in general, which can describe any data movement across any system. For the broader concept, see our blog on what data lineage is and why it matters. ETL lineage is specifically about pipeline-level visibility: the orchestration, the transformations, the loads, and the dependencies that connect them.

ELT pipelines, where transformations run inside the warehouse rather than before the load, raise the same questions. Same failure modes, same conditions for trust. The architecture changes. The lineage problem does not.

How lineage shows up at each ETL stage

Lineage is not a single artifact. It looks different at each stage of the pipeline, and most implementations only capture one or two of the three.

Extract

At the extract stage, lineage records what was read from each data source: which tables, which columns, which snapshots of the schema. The metadata here is comparatively easy to collect because source systems usually expose their schemas through standard interfaces.

Transform

The transform stage is where lineage gets hard. This is the stretch where joins, filters, aggregations, and calculations move values around between fields. Capturing those data transformations accurately requires parsing the SQL or transformation logic and tracing each output column back to the source columns that fed it. A parser that misses transformations or maps them incorrectly creates gaps that undermine trust in the whole graph.

Load

At the load stage, lineage records the destination tables and, ideally, the downstream data assets that consume them: dashboards, ML feature tables, exported datasets, derived models. Without downstream coverage, lineage answers only half the question. It tells you where the data came from, but not what depends on it.

A pipeline can have clean extract-stage lineage, perfect load-stage lineage, and a black hole in the middle where the actual transformations happen. That is the most common failure mode, and it is the one most worth designing against.

Across these stages, lineage gets captured three main ways:

Through SQL parsing of transformation logic
Through runtime emission from orchestration layers like Airflow
Through batch ingestion of historical query logs

Most production setups mix all three.

The three conditions ETL lineage has to meet

Most teams implementing data lineage fall short of at least one of these conditions. Lineage that falls short of all three is not really lineage at all.

1. It has to be column-level, not just table-level

Table-level lineage tells you which tables fed which other tables. That is useful for orientation but not for debugging.

When a finance dashboard shows the wrong revenue figure, knowing the data came from a source Postgres table does not help. You need to know which field fed which field, through which SQL expression, at which transformation step. That is column-level lineage. It collapses hours of cross-system investigation into a navigable graph.

Table-level lineage points you to the right neighborhood. Column-level lineage points you to the address.

SQL parsing is how this gets done at scale. A parser walks the queries that define each transformation, traces every join, aggregation, filter, and calculation, and maps the result back to the source fields that fed it. The accuracy of that parser is what determines whether the graph is trustworthy.

2. It has to span every tool in the pipeline

ETL pipelines do not live inside one tool. A typical production pipeline runs through a source database, an orchestration layer like Airflow, a transformation layer like dbt, a warehouse like Snowflake or BigQuery, and a BI layer like Looker or Tableau.

Lineage that covers one layer of that stack cannot answer the questions that actually come up in production. If a column is wrong at the dashboard and your lineage stops at the warehouse boundary, root cause analysis turns into manual investigation across systems.

The version of lineage that holds up in production is multi-hop and cross-platform. It stitches the pipeline together in a single graph so an engineer can trace a downstream symptom back to its upstream cause without switching tools.

Stitching after the fact, by combining exports from each system’s native lineage, sounds reasonable. It rarely holds. Native lineage uses incompatible identifiers across tools, and the joins fail in exactly the ways you would not notice until you needed them. The graph has to be unified by design.

3. It has to reflect what actually ran, not what was documented

Design-time lineage and runtime lineage are different things. ETL pipelines change constantly. Columns get added. Transformation logic shifts. DAGs get modified. Engineers ship.

Lineage that reflects last month’s architecture misleads engineers into making confident, wrong decisions. The first time a developer pulls up a lineage graph, finds the column they need to investigate, and discovers it does not exist in production anymore, the entire graph loses credibility.

Runtime-captured lineage solves this. The orchestration layer emits lineage as pipelines run. The graph reflects what actually happened in the last execution, not what someone documented when the pipeline was built.

The first time an engineer pulls up the lineage graph and finds a column that has been gone from production for two weeks, you have lost them. After that, every graph reading happens with a footnote in their head.

For systems where periodic updates are sufficient, batch ingestion via query log analysis is a reasonable middle ground. The point is not that everything has to update in real time. It is that the lineage graph cannot represent design intent. It has to represent execution reality.

What changes when ETL lineage clears all three

Lineage that meets every condition changes what is possible operationally. It also makes impact analysis tractable before changes ship. Engineers can see every downstream dashboard, model, and pipeline that depends on a schema or transformation, so a what-if check stops being speculative and starts being definitive.

In IDC’s Business Value Study of DataHub Cloud, customers reported 75% more datasets with mapped lineage, 48% fewer data-related outages, and 58% faster resolution when outages did happen.

The pattern across those numbers is the same. When lineage reflects the reality of what is running across the whole pipeline at the column level, data engineers spend less time investigating and more time fixing.

What complete ETL lineage looks like in DataHub

DataHub addresses each of the three conditions across the data ecosystem structurally, not as a layer added on top of a catalog.

Column-level lineage through SQL parsing

DataHub automatically extracts field-to-field dependencies through SQL parsing during metadata ingestion. The parser traces each column from its source table through every join, aggregation, and calculation to its downstream consumers across Snowflake, BigQuery, dbt, and other systems. In the Lineage graph view, engineers can drill from the table view into individual column dependencies, or isolate a single column’s full path through the pipeline.

One graph across every system

DataHub’s metadata graph connects lineage across heterogeneous systems by design, not by stitching exports together. A complete pipeline path from a source database through an Airflow DAG, a dbt model, a Snowflake table, and a Looker dashboard is navigable in one view. Engineers can expand upstream and downstream dependencies at any node without switching tools. The graph scales to millions of entities, backed by a 99.5% uptime SLA.

This is what made the difference for Chime, whose stack runs Snowflake, Spark, Flink, Airflow, and Protocol Buffer schemas.

My favorite part about DataHub is the lineage because this is one really easy way of connecting the producers to the consumers. Now the producers know who is using their data. Consumers know where the data is coming from. And it is easier to have accountability mechanisms.
Sherin ThomasSoftware Engineer, Chime

Pipeline lineage alongside dataset lineage

DataHub models pipeline-level relationships, including Airflow DAG task dependencies, in the same graph as dataset lineage. The pipeline entity page shows a visualization centered on the pipeline with its composite tasks, plus the upstream and downstream data dependencies for each task.

Runtime and design-time capture, side by side

For pipelines where freshness matters, the DataHub Airflow listener and OpenLineage protocol emit lineage at execution time. For systems where periodic updates are enough, batch ingestion via query log analysis pulls lineage from historical query logs on a schedule. Teams mix both approaches per system, and the unified graph combines the sources.

When lineage earns its place

Lineage in ETL is not a documentation exercise. The three-condition test is the bar: column-level, cross-platform, captured at runtime. Lineage that meets it changes what your team can do when something breaks. Lineage that does not, just decorates the catalog.

FAQs

Data lineage in ETL is the record of how data moves and transforms across the extract, transform, and load stages of a data processing pipeline. It traces each field from its source, through every transformation step, to its downstream destinations like dashboards and ML models. Useful ETL lineage captures dependencies at the column level, spans every tool in the pipeline, and reflects what actually ran rather than what was documented.

ETL is a process for moving and transforming data. It stands for extract, transform, load, and it describes how data gets from source systems into a destination like a data warehouse. Data lineage is the record of what that process did, the field-level history of how each piece of data moved and changed along the way. ETL is the activity. Lineage is the audit trail.

In ETL, transformations happen before data lands in the destination. In ELT, transformations happen inside the destination, usually the warehouse. The architecture is different, but the lineage questions are the same: where did each field come from, how was it transformed, and where is it consumed. Lineage that meets the three-condition test holds up in both.

Table-level lineage tells you which tables feed which other tables. That is enough for orientation but not for debugging. When a downstream metric is wrong, you need to know which specific column produced the bad value and which upstream field fed it through which transformation. Column-level lineage collapses what would otherwise be hours of cross-system investigation into a navigable graph, and it is what makes data accuracy auditable at the field level.

Runtime lineage is emitted by the orchestration layer as the pipeline runs. In DataHub, this happens through the DataHub Airflow listener and the OpenLineage protocol, which capture lineage at execution time rather than from static documentation. The result reflects what actually happened in the last pipeline run, not what someone documented when the pipeline was built.

Yes, but only if the graph is unified by design. Most data lineage tools that promise multi-platform coverage stitch together exports from each tool’s native lineage, which rarely holds up because the identifiers and schemas differ across systems. A lineage platform built on a shared metadata graph can connect a source database, an orchestrator like Airflow, a transformation layer like dbt, a warehouse like Snowflake, and a BI tool like Looker into one navigable view. That is what makes consistent data quality tracking possible across the full pipeline.

Yes. It provides the audit trail that data governance programs and regulatory compliance frameworks like GDPR, SOC 2, and HIPAA require. In an ETL context, lineage shows how sensitive fields enter the pipeline, where they get transformed, and which downstream consumers they reach. Without it, compliance teams reconstruct data flows by hand. With it, the trail is already there.

Three failure modes dominate. The graph covers tables but not columns, so debugging stops at the wrong granularity. It covers one or two layers of the pipeline but not the whole stack, so investigation crosses tool boundaries by hand. Or it reflects design intent rather than execution reality, so it points to columns that no longer exist in production. Lineage that fails any of these tests fails at the moment it was supposed to help.