AI-Generated Documentation and Context Propagation

The real problem with enterprise data documentation isn’t that teams don’t want to write it. It’s that documentation decays faster than humans can maintain it.

AI documentation has become the go-to solution for the cold-start problem: The reality that most data assets never get documented at all because writing descriptions for thousands of tables is nobody’s idea of a productive afternoon. And the tools have gotten good enough that the output is genuinely useful, not the generic placeholder text that early attempts produced.

But here’s the part that gets less attention: Generation alone is a dead end. You can produce a perfect description for every column in your warehouse, and within weeks, most of that context will be stranded at the source while downstream consumers still have no idea what they’re looking at. The missing half of the equation is context propagation—the ability to automatically push documentation and classification labels through your lineage graph so context follows data wherever it goes.

The documentation decay problem

If you’ve worked with enterprise data for more than a few months, you know the pattern:

• Someone launches a documentation initiative
  
• The team writes descriptions for the 50 most important tables
  
• Everyone feels good for about six weeks
  
• Then new tables get created, schemas change, pipelines get refactored, and the documentation quietly rots
  
• Within a quarter, the catalog is a mix of accurate descriptions, outdated ones, and thousands of assets that never got documented in the first place
  

This isn’t a discipline problem. The math just doesn’t work. A mid-size data organization might have tens of thousands of tables and hundreds of thousands of columns across Snowflake, Databricks, dbt, and a handful of SaaS tools. Even a dedicated documentation team can’t keep up with the rate of change, let alone start from scratch.

The cost of this gap goes well beyond aesthetics:

• Analysts waste hours trying to figure out what a column named status with values 0 through 3 actually means
  
• Data engineers rebuild assets that already exist because they can’t find or trust what’s already there
  
• Governance teams can’t enforce policies against data they can’t describe.
  

And increasingly, AI agents operating against your data estate can’t reason about assets that lack context. An undocumented table is effectively invisible to an AI agent, no matter how valuable the data inside it might be.

The result is a cold-start problem at organizational scale. Most assets never get documented. The ones that do go stale within months. And the humans who could fix it are already fully allocated to work that actually ships.

What AI-generated documentation actually does

AI-generated documentation addresses the cold-start problem by producing table and column descriptions automatically, at scale. But the quality of what gets produced depends entirely on what signals the AI can access.

Early approaches relied on column names and data types alone, which produced output that was only marginally more useful than the column name itself. A description that says “customer_id is an identifier for customers” adds nothing.

The shift that made AI documentation genuinely useful was giving the model access to holistic context. In DataHub, AI documentation draws on multiple signals to produce each description:

• Dataset and column names and types provide the baseline, but never the full picture
  
• Lineage relationships show where data came from and where it flows downstream, revealing transformation logic that pure naming can’t
  
• Profiling statistics reveal what the data actually looks like in production
  
• Existing documentation on related upstream and downstream assets provides inherited context
  
• Glossary terms supply the organization’s business vocabulary
  
• Query-derived signals (where available) help ground descriptions in how the data is derived

When AI documentation has access to this full context signal, the output reflects how data actually behaves rather than what someone guessed when they named the column. A column called amt that traces back through a revenue pipeline, contains decimal values averaging $47.50, and appears in three executive dashboards gets a fundamentally different description than the same column name in a test table.

The goal isn’t perfect documentation on every asset. It’s useful context on every asset. AI gets you from zero to genuinely helpful at a scale that manual processes never could, and humans stay in the loop where it matters most.

In DataHub, AI-generated descriptions are marked with a visual indicator (a sparkle icon) so teams can immediately distinguish content that hasn’t been reviewed from human-approved documentation. This preserves trust in the catalog. Nobody has to wonder whether a description was vetted or auto-generated because the interface tells them.

Custom instructions add another layer of control. Organizations can configure the AI to use their specific terminology, standards, and conventions, so output reads like it was written by someone who actually works there rather than by a model that’s guessing at your domain vocabulary.

Pinterest used DataHub’s AI documentation to document its data at scale. The team generated table and column descriptions automatically using lineage, existing docs, glossary terms, and query-derived signals (where available). This cut manual documentation effort by about 40%, with humans staying in the loop for the most critical assets. It’s a practical tradeoff: your most important tables still get expert attention, but the long tail of assets that would otherwise sit undocumented now has meaningful context that both people and AI agents can use.

Generation without propagation is a dead end

Here’s where most AI documentation initiatives stall.

You’ve generated descriptions for your source tables. Coverage is up. The team is happy. But then an analyst opens a downstream dashboard table three transformations removed from the source, and there’s nothing there. The documentation you generated sits at the origin, and nobody copied it downstream. Why would they? Across hundreds of tables and thousands of column-level relationships, manual propagation is the same unscalable busywork that AI documentation was supposed to eliminate.

The result is a coverage plateau. Your most important upstream assets are documented, but the downstream tables and views that analysts and AI agents actually interact with remain undescribed. Context exists in your catalog, but it doesn’t reach the places where people need it.

This is the gap between generating documentation and delivering context. Generation fills in the blanks. But without a way to propagate context through lineage, those answers stay stranded at the source.

How context propagation works

In DataHub, context propagation operates through two complementary mechanisms.

Documentation propagation

Documentation propagation automatically pushes column descriptions downstream based on column-level lineage and sibling relationships. When a column is documented at the source, that description follows lineage to every downstream asset that inherits from it.

This means a description written once at the Bronze layer appears everywhere the column shows up in Silver and Gold layers, without anyone manually copying it. For organizations running hundreds of transformation steps between raw ingestion and analyst-facing tables, this is the difference between documenting once and documenting never.

Propagated descriptions are visually marked with a thunderbolt icon, so users can immediately see which descriptions were propagated versus authored directly. A tooltip shows the origin asset and any intermediate hops, giving full transparency into where the description came from.

For teams running DataHub Cloud, historical backfilling means you don’t have to start fresh. When you enable propagation, existing descriptions are pushed retroactively across your historical lineage. And if a propagated description turns out to be wrong, propagation rollback lets you undo any change.

Glossary term propagation

Glossary term propagation works alongside documentation propagation but serves a different purpose. Instead of pushing descriptions, it propagates classification labels (glossary terms) across columns and assets based on downstream lineage and sibling relationships.

The use case here is standardization. When a business concept like “Advertiser ID” or “Customer Lifetime Value” appears across dozens of tables under slightly different column names, glossary terms create a common language that links them. Apply the term once at the source, and propagation ensures every downstream column representing that concept carries the same classification.

Pinterest took this approach to standardize business concepts across its data estate. The team analyzed join patterns in query logs to identify columns representing the same entities, then propagated glossary terms automatically. The result: more than 40% of their columns were classified without manual tagging.

The integrated workflow: Generate, review, propagate

AI documentation and context propagation aren’t two separate features you turn on independently. They’re stages in a single workflow that compounds over time.

The cycle works like this:

• AI generates descriptions using lineage, profiling data, and related metadata
  
• Humans review and refine the most critical assets, approving or editing AI-generated content
  
• Documentation propagation pushes those approved descriptions downstream through column-level lineage
  
• Glossary term propagation standardizes business concepts across the estate
  
• And as new assets are created and lineage extends, the cycle repeats automatically
  

The compounding effect is the key. Every description you write or approve at the source multiplies its value through propagation. Document 100 source columns, and you might cover 500 downstream columns for free. Add glossary terms to your most important business concepts, and classification reaches every table that touches them. Over time, the ratio of effort-to-coverage improves continuously as your lineage graph extends and new assets automatically inherit context from their upstream sources.

This compounding matters even more when you consider how data teams actually work. Most organizations have a relatively small number of canonical source tables that feed a much larger number of downstream transformations, aggregations, and reporting views. Documentation propagation follows that same fan-out pattern. A single well-documented source column might propagate to dozens of downstream views, each of which would have required its own manual documentation under the old model.

For organizations investing in AI agents, whether through MCP servers, data analytics agents, or internal copilots, this workflow is foundational. An AI agent’s usefulness scales directly with the richness of the context available to it. Propagated documentation and glossary terms are what transform a data catalog from a static reference into the semantic backbone that AI agents need to find, understand, and correctly use your organization’s data.

The practical tradeoff is one most data teams can live with: Your most important tables get expert human curation. The thousands of others that would otherwise sit undocumented get AI-generated descriptions that are good enough to be useful. And propagation ensures that even the long tail of downstream assets inherits whatever context exists upstream. It’s not perfect documentation everywhere. It’s useful context everywhere, which is a dramatically better outcome than the alternative.

FAQs