2024 in Review; 2025 in View: Metadata Management Trends Shaping the Future of AI and Data Governance

As I watched the Metadata & AI Summit 2024 wrap up at the end of October, I couldn’t help but reflect on the extraordinary transformation happening in the tech world. During the ten years I spent at LinkedIn, I personally navigated a shift from solving lower level storage, retrieval, and computation problems to higher level semantics, consistency and governance problems. It became clear to me that as organizations mature in how they work with data, they strive for higher ground, with data management, quality and compliance taking center stage. When I founded DataHub to solve some of these problems, I knew we were addressing a fundamental challenge in how organizations manage their data and, increasingly, their AI assets. In the last two years, it seems like we’ve sped through a wormhole of innovation, like a teenager going through a sudden growth spurt; but just like the gangly youth — we may need to take some mindfulness lessons to truly mature as an industry.

Enterprises and business leaders today understand that they must shift to incorporate AI into their fabric. But the core challenge they face is the complexity involved. Over the years, I’ve watched organizations grapple with data and AI supply chains that have grown impossibly intricate. AI engineers puzzle over whether their models are underperforming due to algorithmic issues or missing data, while compliance officers struggle with seemingly basic questions like whether models were trained using personal information. Only one constant remains: chaos. Traditional governance approaches have failed — they’re resource-intensive, disconnected from daily workflows, chronically underfunded, and overly dependent on manual intervention.

But here’s where it gets exciting: generative AI can be the solution to its own governance challenges. I believe AI can automate up to 90% of governance activities. The key is a human-in-the-loop approach, especially for high-stakes outcomes like compliance. Through DataHub, we’re building a “high-fidelity Metadata Graph” that brings order to this chaos. When it comes to utilizing AI for governance, our approach is simple yet powerful:

1. Propose: Let AI suggest classifications and documentation
2. Approve: Have the right humans validate for safety
3. Monitor and Enforce: Use automated monitoring and enforcement driven by these governance primitives

Though the AI age brings new opportunities for governance, it also brings a whole new set of challenges. Unlike traditional data governance, we now need ways to represent entirely new types of assets that pertain to the AI supply chain, extreme versioning capabilities, AND systems that can handle constant metadata production.

We’re evolving DataHub to address these new needs; creating a versioned data and AI Asset Registry that can track everything from models and features to prompts and functions and can handle extreme scale, volume and velocity of metadata. Our goal is to enable a unified governance approach for data and AI, and a consistent source of truth that’s scalable and serves multiple stakeholders.

We’ve also learned some critical lessons along the way. Our first attempts at AI-assisted documentation were rough–we over-indexed on structural information and missed the human touch. But we’ve improved. Now, we can generate documentation that provides compliance insights, content context, and actual use cases.

The most important insight? Good governance isn’t about restricting; it’s about enabling innovation safely. We’re building open interfaces for generative AI that help organizations:

• Automate documentation
• Implement intelligent classification
• Monitor data quality
• Ensure compliance

With technology continuing to advance at a dizzying pace, 2025 promises to build upon these innovations, pushing the boundaries of what’s possible in metadata management and further shaping the future of data governance. Here are my predictions for what lies ahead.

Metadata Shifts to Watch in 2025

Now that GenAI is stepping out of the “tinkering zone” into production, challenges and opportunities around AI reliability and governance will be top of mind for leaders.

As I look to the near future, I see four trends playing out:

1. The AI wave, powered by generative AI, becomes real: Powered by generative AI, we’re experiencing an unprecedented democratization of artificial intelligence. Service providers like OpenAI and Anthropic, along with open-source models like LLAMA and Mistral, have dramatically reduced the barriers to creating AI-powered experiences. The time to build sophisticated AI applications has compressed from months or years to weeks or even days. But this isn’t just about technological capability — it’s about competitive survival. Organizations are now facing a stark reality: become AI-powered or risk becoming obsolete. According to Gartner, 70% of business leaders believe generative AI will significantly change how they create, deliver, and capture value. This isn’t speculation; it’s happening right now. Companies are rushing to integrate AI not as a nice-to-have, but as a critical strategic imperative. The accessibility of AI has turned what was once the domain of tech giants into a playing field where even smaller organizations can leverage cutting-edge technology to drive innovation, efficiency, and growth.
   
2. AI brings both challenges and opportunities for governance: As AI becomes more powerful and pervasive, it simultaneously presents remarkable opportunities and unprecedented challenges for governance teams. On the opportunity side, AI can automate up to 90% of traditional governance activities, from describing assets to finding critical information, dramatically reducing the manual labor that has long plagued compliance efforts. We can now use AI to predict sensitivity levels, generate high-quality documentation, implement intelligent classification, and monitor complex compliance requirements with unprecedented speed and accuracy. However, these opportunities come with significant challenges. AI introduces a new layer of complexity to metadata management, requiring extreme versioning capabilities, systems that can handle constant metadata production, and ways to represent entirely new types of AI assets. Governance teams must now track not just data, but models, features, prompts, functions, and embeddings, each with its own intricate lineage and compliance requirements. The stakes are high: organizations need to balance innovation with safety, ensuring that AI systems are not just powerful but also responsible, transparent, and aligned with emerging regulatory standards. This isn’t just a technical challenge, but a strategic imperative that requires a fundamental rethinking of how we approach governance in the age of generative AI.
   
3. A unified metadata platform is crucial: A unified metadata platform isn’t just a technological luxury. It’s an absolute necessity. The complexity of modern data ecosystems has reached a point where traditional, fragmented approaches to metadata management are no longer viable. A truly unified platform serves as a single source of truth, bridging the gaps between different teams, technologies, and data assets. Through our work with DataHub, we’ve seen how such a platform can transform chaos into clarity, providing a comprehensive view that spans people, data, and code. It’s not just about storing metadata; it’s about creating a high-fidelity metadata graph that enables discovery, observability, and federated governance. Such a platform must be flexible enough to handle traditional data assets and the new wave of AI artifacts — from machine learning models and features to prompts and embeddings. Crucially, it needs to serve multiple stakeholders simultaneously: data engineers, AI researchers, compliance teams, and business users. The platform becomes the connective tissue of an organization’s data strategy, providing context, ensuring compliance, enabling discovery, and ultimately accelerating innovation. Without this unified approach, organizations risk creating siloed, inconsistent, and ultimately unreliable metadata ecosystems that can severely hamper their ability to leverage data and AI effectively.
   
4. The future has to be Open and Collaborative: The future of AI governance cannot be a closed, proprietary endeavor. It must be built collaboratively, with transparency and collective intelligence at its core. Through the DataHub project, we’re demonstrating that open source is not just a development model, but a powerful approach to solving complex technological challenges. By creating an open platform, we invite contributions from diverse perspectives, whether from enterprise leaders at companies like Apple, Netflix, Optum, or Visa, or from individual developers and researchers around the world. This collaborative approach allows us to rapidly iterate, stress-test our assumptions, and develop solutions that are more robust and adaptable than any single organization could create in isolation. Our community of over 12,000 members brings different experiences, use cases, and insights, helping us evolve the platform beyond what we initially imagined. Open interfaces for generative AI mean that anyone can contribute, extend, and customize the platform to meet their specific needs. We’re doing more than just building a tool; we’re cultivating a movement that recognizes that the challenges of AI governance are too complex and too important to be solved by any single entity. By building openly, we’re creating a shared infrastructure that can help organizations worldwide navigate the complex landscape of data and AI governance, ensuring that innovation is balanced with responsibility, transparency, and collective wisdom.

2024 Was Just the Beginning

The trends of 2024 have laid the groundwork for a transformative 2025. As AI continues to reshape how we approach data governance, the coming year promises even more opportunities for businesses to leverage automation, ensure data quality, and embrace decentralized, cloud native platforms. The role of metadata in powering data discovery, governance, and trust has never been more critical. As we move into the new year, the metadata landscape will continue to evolve, offering new challenges and exciting possibilities for organizations across all industries.

My invitation to the tech community is simple: Join us. Help build a platform that enables all of us to unleash AI’s potential while mitigating its risks.