Visa Scales Data Governance

Note: This story was originally published March 2024.

Challenge

As the global leader in digital payments, Visa processes massive volumes of sensitive data daily. Security, trust, and seamless user experiences are non-negotiable—which makes scalable, effective data governance essential. For Visa, data governance is not merely a matter of compliance; it is a fundamental aspect of responsible data management. 

Visa built a custom metadata platform to support a metadata-first data strategy. While this system helped establish foundational governance practices, it became a resource-intensive bottleneck. Maintaining it demanded constant engineering attention and limited the company’s ability to scale governance with the business.

Visa faced three core challenges:

1. Managing classifications at scale: They needed to standardize attribute-level classifications, definitions, and access policies, without offloading business metadata ownership to data engineers.
2. Capturing high-quality, validated metadata: Needed to encourage data owners to provide high-quality and validated annotations of data assets.
3. Managing copies of data across environments: Datasets are frequently replicated across physical environments. This complexity made it difficult for users and stewards to navigate and discover the data they needed, creating inefficiencies in data access and management.

The challenge was particularly difficult to solve given the scale at which Visa operates.

“We need to figure out how, across thousands, and in some cases hundreds of thousands of datasets and millions of columns, how to get all this connected and appropriately propagated through to the catalog to consumers.”

— Jean-Pierre Dijcks, Senior Director of Product Management, Visa

Solution

Visa chose DataHub after a comprehensive evaluation of leading metadata platforms. What stood out: DataHub’s API-first architecture, feature-rich capabilities, and user-friendly interface.

Visa implemented several capabilities to address their top priorities:

• Business Attributes Model: Visa developed a logical model to centralize and maintain crucial business information owned by data stewards and subject matter experts. This approach ensured that critical business context remained accessible and well-managed.

Note: Business Attributes are available as of DataHub v0.13.3!

• Structured Properties: DataHub’s Structured Properties approach is helping Visa’s data platform team a) streamline metadata management and b) improve the developer experience while facilitating more efficient integration of API data into applications.
• Logical Datasets: “Where is the data? Where does it live in a physical context? What does it mean in the business context?” These are the kinds of questions Logical Datasets can help answer. Visa developed a Logical Dataset capability to connect replicated tables across physical environments, aimed at establishing a more scalable governance model where data products, contracts, and definitions are interconnected seamlessly.  

Visa also embraced an “invisible catalog” approach, using DataHub’s robust API capabilities to embed powerful metadata flows into existing tools and processes.

“We have a lot of users that never interact with the UI of our catalog. But we have a ton of data engineers and users that interact through APIs and to various other tool integrations.”

— Jean-Pierre Dijcks, Senior Director of Product Management, Visa

Impact

Visa replaced its bespoke catalog with a scalable metadata platform that aligns governance with how the business actually works.

Key outcomes include:

• Achieved scalable data governance, managing common classifications, definitions, and access policies across its extensive system without overburdening data engineers
• Improved data quality through the Business Attributes model, ensuring end-users receive curated, accurate, and relevant information in real-time
• Enhanced AI capabilities, particularly in critical fraud detection capabilities by leveraging better metadata management to improve underlying data quality
• Streamlined metadata management by implementing DataHub’s API-first approach to create an ‘invisible catalog’ that integrates metadata capabilities into existing tools and workflows
• Freed up engineering resources by replacing their custom-built data catalog solution with DataHub, allowing technical teams to focus on value-driving activities rather than catalog maintenance
• Simplified data discovery through the implementation of Logical Datasets, making it easier for users and stewards to discover and manage datasets across multiple environments