AWS Data Lineage: From Native Capture to Your Entire Stack

AWS captures lineage natively: Amazon DataZone draws it from Glue and Redshift, Amazon SageMaker Catalog covers machine learning models, and an OpenLineage-compatible API handles other sources.

But that native capture only follows AWS’s own services, and most data does not stay within them. DataHub’s data lineage builds on what AWS already captures, extending the same view across the rest of your stack, native or not.

What lineage does AWS capture natively?

If your data already lives in AWS, you already have a head start on data lineage, though it comes from a few services working together rather than one. AWS Glue holds the metadata, the catalog of your tables and their schemas. Amazon DataZone sits above it for governance and business glossary, drawing lineage from Glue and Redshift so teams can trace how data moves. Amazon SageMaker Catalog captures lineage for machine learning models. Once those sources are connected, capture runs automatically, with no manual mapping to maintain.

That native capture is pretty solid. According to the Amazon DataZone documentation, native lineage:

• Captures automatically from Glue and Redshift
  
• Surfaces column-level detail wherever source-column information is available
  
• Records each event as a versioned node so you can compare how a table or job changed over time
  

The same automatic capture for Glue crawler sources is walked through step by step on the AWS Big Data Blog. The result is a visual lineage graph you can traverse upstream and downstream, expand to the column level, and filter down to a single field.

Inside AWS, that is enough for the everyday work lineage exists to do. The DataZone documentation frames it around four jobs:

• Understanding where data came from
  
• Assessing the impact of a pipeline change before it ships
  
• Tracing a data-quality issue back to its source at the column level
  
• Demonstrating governance by showing where sensitive data such as personally identifiable information (PII) is stored and how it moves downstream.
  

For everything that happens outside the catalog, AWS exposes an OpenLineage-compatible API. Transformations in Amazon S3, Spark jobs in AWS Glue, and other steps can push lineage events programmatically, so they appear alongside the automatically captured graph. That covers a lot of ground, and for teams working entirely inside AWS it answers the core questions lineage exists to answer: where did this data come from, what changed it, and what breaks downstream if I touch it.

Where do Glue, DataZone, and SageMaker lineage stop?

Native lineage on AWS is not one feature. Depending on the use case, teams lean on AWS Glue for the metadata catalog, Amazon DataZone for lineage and governance, or Amazon SageMaker Catalog for model lineage. Whichever you use, the reach ends in the same two places: the services AWS does not operate, and the gaps between the ones it does.

Non-native tools and services

The moment data crosses into something AWS does not operate (a BI tool like Tableau or Looker, a warehouse like Snowflake, or a reverse-ETL sync), lineage depends on OpenLineage events you instrument and maintain yourself. That holds even when the tool runs on AWS, since native lineage follows AWS’s own services rather than everything hosted in your account. Where that wiring is missing, the trace simply stops.

Picture a Redshift table that feeds a dashboard in Tableau. Inside DataZone, the lineage is clean right up to the edge of Redshift. But Tableau is not a native AWS service, so the dashboard, and the analysts who depend on it, sit on the far side of a gap. When that dashboard breaks, the trace that would tell you why ends at the boundary.

The gaps between AWS services

The second limit is assembly. Native lineage is captured service by service, and each surface has its own story.

Raw objects in Amazon S3 do not produce lineage on their own. They have to be registered as Glue tables first, through a crawler or your infrastructure-as-code, before DataZone can correlate them. Getting consistent column-level lineage out of a Glue ETL job means running AWS Glue 5.0 or later and configuring the OpenLineage integration correctly, with the right job parameters and permissions. There are scale limits too: a DataZone lineage run fails after 100 tables, so large crawls have to be split. None of this is unreasonable. It is simply work, repeated per service, that has to stay correct as data pipelines evolve.

Within AWS, native lineage does its job well. It is also scoped to the services AWS operates and assembled service by service, and that puts the gaps exactly where multi-platform stacks operate: at the tools AWS does not run, and between the ones it does.

Extending AWS lineage across your entire stack

A dedicated lineage layer does not replace what AWS captures. It connects to the same services, unifies their lineage in one graph, and carries the trace past the boundary.

DataHub integrates directly with Glue, Redshift, S3, Athena, and SageMaker, so a single framework covers what native lineage otherwise splits across Glue, DataZone, and SageMaker Catalog:

• AWS Glue and Amazon Redshift: Column-level lineage extracted directly from Redshift query history and Glue job logic through SQL parsing, mapping how individual columns flow through transformations.
  
• Amazon S3: Schema inference and metadata ingestion, so data lakes are cataloged and their structure, down to column data types, is visible.
  
• Amazon Athena: Metadata ingestion with table-level lineage through Glue catalog relationships.
  
• Amazon SageMaker: Machine learning metadata, including training jobs, registered models, and feature group relationships.
  

The game-changer: 100+ connectors

With 100+ connectors, the same graph that captures your Glue-to-Redshift column flow keeps going into Snowflake, BigQuery, dbt, Tableau, and the rest of your estate, so a column trace does not dead-end when it leaves AWS’s own services.

Capabilities that turn lineage into an operational tool

Around that unified graph sit the capabilities that turn lineage from a diagram into an operational tool:

• Impact analysis: Bidirectional traversal that shows an engineer the full upstream and downstream blast radius, at table and column precision, before a schema change ships.
  
• Metadata propagation: Tags, descriptions, and ownership assigned upstream, pushed down to every downstream asset automatically, so you label something once and see it everywhere.
  
• Metadata tests and usage tracking: Detection of unused or duplicate tables, confirming that an apparent orphan truly has no downstream consumers before anyone deprecates it.
  
• Scheduled ingestion: A frequency you configure that keeps lineage current as pipelines change, with no streaming pipeline to stand up.
  
• Visual lineage explorer: An interactive graph that lets you visualize lineage across the whole estate, filterable by time, owner, or platform to narrow it to the slice that matters.
  

Let’s make it real: An engineer about to drop a column in Redshift needs the full downstream blast radius before the change ships. Native lineage shows the part of that radius that lives in AWS. If the column also feeds a Looker model or a Snowflake table, those consumers sit outside the view, and the change goes out blind to them. Bidirectional, cross-platform impact analysis closes that gap.

When a pipeline breaks, a report looks off, or an auditor asks where data came from, the full-path view is what each team reaches for:

• Data engineers see what broke and why across the whole path, not just the AWS leg.
  
• Analysts know which reports they can trust because they can see what feeds them.
  
• Compliance teams trace sensitive data back to its source even when the journey runs through systems AWS does not operate.
  

How Adevinta unified lineage across AWS and beyond

Adevinta, which runs a network of local and global online marketplaces, has exactly the kind of stack this problem describes. Its data estate spans:

• AWS:
  
  • Amazon S3
    
  • Amazon Athena
    
  • AWS Glue
    
  • Amazon Redshift
    
• Non-AWS:
  
  • Databricks
    
  • BigQuery
    
  • Snowflake
    
  • dbt
    
  • Looker
    
  • Tableau
    
  • Datadog
    

Local and global teams each brought their own tools, and the result was a fragmented landscape where data was hard to find and harder to trace.

Adevinta built a centralized catalog on DataHub, ingesting metadata and lineage from across that mixed stack on an hourly cadence. The outcome was a single interface spanning AWS and non-AWS systems alike, with more than 65,000 data entities made searchable, lineage visualized across tools rather than trapped inside each one, and clearer paths for teams to find, evaluate, and share data across the organization.

Our data catalog is a functional part of our DataHub, which is our main product to explore, access, store and share data within Adevinta.

Oscar OmpreProduct Manager, Adevinta

The native lineage in Adevinta’s AWS services did real work. Unifying it with everything else is what made the estate navigable.

Deploying DataHub on AWS

Adding a lineage layer should not mean moving your data or loosening your security posture. DataHub Cloud is hosted on AWS infrastructure, with native Amazon Bedrock integration powering its AI features. Ingestion can run inside your own virtual private cloud (VPC), so credentials never leave your environment.

DataHub is available on AWS Marketplace, which means a purchase counts toward your AWS committed spend.

FAQs