HashiCorp Streamlines Their Data Discovery Chaos
“Before implementing DataHub, we received a lot of different inquiries about data … Since implementing DataHub, we've mostly found that we don't experience this challenge anymore. A lot of these ad hoc inquiries are down to maybe one at most per day.”
- Challenge: New analysts struggled with long onboarding times due to limited data visibility, scattered documentation, and difficulty identifying data owners
- Solution: Implemented DataHub as a centralized data dictionary with integrated ownership tracking and automated documentation, seamlessly connecting with existing tools like Tableau, dbt, and Snowflake
- Impact: Achieved streamlined data discovery across the business, reduced ad hoc inquiries to a maximum of one per day, and improved governance visibility with clear data lineage for developers
Note: This story was originally published June 2024.
Challenge
HashiCorp, an infrastructure cloud company that helps organizations automate multi-cloud and hybrid environments, faced a major challenge with onboarding new data analysts: lack of data visibility.
New analysts often spent hours combing through Slack threads and email chains just to find the information they needed. Even when documentation existed, it was scattered across multiple systems, making it hard to locate reliable answers. This led to a constant stream of ad hoc questions directed at existing analysts, creating a frustrating bottleneck that slowed down analytics workflows and delayed time-to-value for new hires.
To address the issue, the team aligned on a clear goal: reduce the time it takes for analysts to ramp up and start delivering impact.
“As a new analyst, it’s difficult to understand what data and analytics are available and who owns them.”
Nathan SiaoData Analyst, HashiCorp
Solution
HashiCorp’s Data Analytics team turned to DataHub to address their challenges, drawn by several key capabilities that aligned with their needs:
- Centralized data discovery: DataHub provided a unified view of their data landscape, making it much easier for analysts to find and understand available datasets
- Seamless integration: The platform’s compatibility with HashiCorp’s existing tools was crucial
- Open source flexibility: As an open-source platform, DataHub allowed the HashiCorp team to continuously customize and improve the system to meet their unique requirements
The implementation focused on two critical areas:
- Ownership management: The team systematically addressed fundamental questions like “Who is responsible for business logic? Who owns the ingestion of the dataset?” They defined different types of ownership (technical, business, and system) and used DataHub ingestion transformers to assign and document ownership clearly
- Streamlined documentation: The team leveraged DataHub’s integration with dbt docs to automatically import existing documentation, then used the Python SDK to programmatically update documentation for non-dbt data assets, effectively centralizing all documentation in one accessible location
Impact
Implementing DataHub has led to major improvements for HashiCorp’s analytics and data engineering teams, empowering both new and existing team members to work more efficiently and independently.
Key outcomes include:
- Near-elimination of ad hoc inquiries frees up time for high-impact work
- Faster analyst onboarding by enabling self-serve access to centralized documentation
- Improved data discovery by consolidating scattered documentation into a single source of truth
- Stronger data governance for both technical and business users who now have clearer visibility into data relationships and dependencies
- More efficient developer workflows through enhanced lineage tracking
“We’ve experienced a streamlining of data discovery across the business, making it easier for analysts to understand available data and analytics, while developers and administrators gained clearer visibility from a governance perspective.”
Nathan SiaoData Analyst, HashiCorp