Deutsche Telekom Calls the Experts to Streamline Data Discovery

Note: This story was originally published October 2024.

Challenge

Deutsche Telekom Digital Labs was drowning in data complexity. The company was managing hundreds of terabytes of data monthly across thousands of datasets, serving 11 countries with diverse regulatory requirements. 

The organization faced an endless stream of questions directed to their Data Team, including fundamental queries about data acquisition, content, access, ownership, utilization, and dashboard availability. This created bottlenecks and inefficiencies across multiple organizational roles.

Despite investments in building “the best data architecture” and tooling, the team realized something critical was missing.

“We realized that there was something missing because the speed at which we were generating the output was not enough for the business.“ 

— Shishidhar Singhal, Director of Engineering, Deutsche Telekom Digital Labs

Solution

After evaluating multiple metadata catalog solutions, Deutsche Telekom Digital Labs selected DataHub for its easy-to-use UI with quick search, easy onboarding, automatic metadata scanning, diverse capabilities like end-to-end data lineage, strong community support, and open-source availability.

Their implementation approach included:

Technical deployment:

• Used Helm Chart for efficient deployment and management
• Established metadata ingestion connections from seven core systems: Nifi, Athena, Redshift, Tableau, Redash, Kafka, and Opensearch
• Completed metadata for data assets across the organization

Organizational structure:

• Defined a comprehensive business glossary with standard business terms linked to data assets
• Implemented Domains in DataHub to organize data assets into logical collections aligned with business units
• Established end-to-end lineage tracking across all systems

Impact

With DataHub, the Deutsche Telekom Digital Labs team has seen significant improvements across multiple dimensions of their organization.

Key outcomes included:

• Eliminated data discovery bottlenecks that generated 4-5 daily calls to data engineers
• Reduced onboarding time from weeks to days for new team members
• Accelerated incident resolution from days to minutes
• Enabled GenAI analytics platform using DataHub metadata as LLM context
• Automatic PII detection flags fields where PII information is present
• Faster impact resolution with repository links for data pipelines
• Cost savings from identifying and removing dead pipelines and transformations