HashiCorp Streamlines Their Data Discovery Chaos 

Note: This story was originally published June 2024.

Challenge

HashiCorp, an infrastructure cloud company that helps organizations automate multi-cloud and hybrid environments, faced a major challenge with onboarding new data analysts: lack of data visibility. 

New analysts often spent hours combing through Slack threads and email chains just to find the information they needed. Even when documentation existed, it was scattered across multiple systems, making it hard to locate reliable answers. This led to a constant stream of ad hoc questions directed at existing analysts, creating a frustrating bottleneck that slowed down analytics workflows and delayed time-to-value for new hires.

To address the issue, the team aligned on a clear goal: reduce the time it takes for analysts to ramp up and start delivering impact.

“As a new analyst, it’s difficult to understand what data and analytics are available and who owns them.”

 — Nathan Siao, Data Analyst, HashiCorp

Solution

HashiCorp’s Data Analytics team turned to DataHub to address their challenges, drawn by several key capabilities that aligned with their needs:

• Centralized data discovery: DataHub provided a unified view of their data landscape, making it much easier for analysts to find and understand available datasets
• Seamless integration: The platform’s compatibility with HashiCorp’s existing tools was crucial 
• Open source flexibility: As an open-source platform, DataHub allowed the HashiCorp team to continuously customize and improve the system to meet their unique requirements

The implementation focused on two critical areas:

1. Ownership management: The team systematically addressed fundamental questions like “Who is responsible for business logic? Who owns the ingestion of the dataset?” They defined different types of ownership (technical, business, and system) and used DataHub ingestion transformers to assign and document ownership clearly
2. Streamlined documentation: The team leveraged DataHub’s integration with dbt docs to automatically import existing documentation, then used the Python SDK to programmatically update documentation for non-dbt data assets, effectively centralizing all documentation in one accessible location

Impact

Implementing DataHub has led to major improvements for HashiCorp’s analytics and data engineering teams, empowering both new and existing team members to work more efficiently and independently.

Key outcomes include:

• Near-elimination of ad hoc inquiries frees up time for high-impact work
• Faster analyst onboarding by enabling self-serve access to centralized documentation 
• Improved data discovery by consolidating scattered documentation into a single source of truth 
• Stronger data governance for both technical and business users who now have clearer visibility into data relationships and dependencies
• More efficient developer workflows through enhanced lineage tracking

“We’ve experienced a streamlining of data discovery across the business, making it easier for analysts to understand available data and analytics, while developers and administrators gained clearer visibility from a governance perspective.” 

 — Nathan Siao, Data Analyst, HashiCorp