DataHub Community Updates: The Feb’23 Low-Down

Hello, DataHub Enthusiasts!

It’s March already (how?!), which means it’s time to share what the DataHub Community has been up to in February.

There’s a LOT that the fantastic Community has done to make DataHub better, easier to use, and all-around amazing.

Let’s get right to it.

Growing Strong: DataHub Community Updates

Our Slack Community now has 6400+ outstanding members, with over 400 folks joining us in February. We continue to have nearly 1,000 active users every week, and our emoji game is still going strong (up 200+ from last month! :D)

If you’re new to your DataHub journey, remember to join our Slack! The Community is incredibly supportive, kind, and eager to help 😊

Upcoming Events in March

DataHub<>BigQuery Workshop

For those just getting with DataHub, we’re hosting an interactive session on BigQuery ingestion. We have limited seats, and it’s happening this Thursday, so remember to RSVP here.

Meet us at Data Council in Austin, March 28–30

The Core DataHub Team is stoked to head to Austin this month for Data Council. We’re packing a ton of content into these three days. Mark your calendar for the following:

• Shirshanka Das’s keynote talk on March 28th: Building a Control Plane for Data
• Paul Logan and I are running a hands-on workshop to help folks get up & running with DataHub on March 28th: URGENT! Help these Pets Find Homes: Working Across Teams in DataHub
• I’m speaking with Kyle Eaton from Great Expectations on March 29th: Teamwork Make the (Open Source) Dream Work: The Power of Cross-Community Collaboration

Lastly, DataHub and Astronomer are excited to invite you to an evening of tacos, beverages, and fun during Data Council. Join us at Lazarus Brewing on Wednesday, March 29th, at 5:30 pm for a night filled with networking opportunities, delicious food, and great company.

RSVP here, and we will send Uber codes to all attendees for transportation to the event.

Pssst: If you still need to sign up for Data Council, use the code DataHub20 to get 20% off your ticket.

Community Case Study: Hurb’s Journey with DataHub

We heard from Patrick Braz & the Hurb Team during the February Town Hall about their DataHub adoption journey. Patrick shared how they use DataHub for easier discovery, security, and better decision-making, with fast-growing assets, new solutions, and growing complexity.

Check it out here:

DataHub + Herb = Data Hurb?

What’s Happening: Roadmap Updates

We launched v0.10.0 in early February, a massive release for us with many additions and improvements. Here’s a quick snapshot of what we shipped:

I’m most excited about

1. Time-aware lineage, which is SUCH a great addition to an already amazing feature.
2. Schema coverage: We’ve added support for JSON schema in this release, and we’ll continue to improve our coverage of all kinds of schemas in the ecosystem.

On the roadmap front, here’s how we’re doing. Things are in motion, and I can’t wait to see how they land 🙂

Progress toward our Q1’23 Roadmap as of Feb’23

Product Updates: Improving and Simplifying DataHub
Improvements to DataHub Search

We’ve improved DataHub’s search experience to implement a scrolling search. We moved on from using an Elastic Search API approach that prevented you from searching past 10,000 search results. A bonus is that this also fixes our long-standing CSV download issue for large search results.

This update makes our Search feature so much more effective, intuitive, and valuable, especially when searching across lineage — where there’s much more complexity within the metadata graph.

Ryan Holstien shows off DataHub’s new and improved Search experience

Redesign of the Queries Tab

You asked, we delivered. The Community needed a way to view, document, and share other commonly used SQL queries (say, those not executed by a daily job but run by analysts in an ad hoc fashion). We needed a curated query set — alongside top or recently executed queries.

Our redesigned Queries tab on the dataset page helps you

• Discover more queries
• Add an element of human curation, and thus
• Pack in more context for better discovery.

John shares how DataHub’s redesigned Queries tab can help with better discovery and context

Unified Redshift Ingestion

Previously, our Redshift ingestion source required configuring two connectors: one for metadata & lineage ingestion and another to extract top queries & usage. We have unified the Redshift ingestion — and brought along a few more changes you’ll like.

Here’s Tamás telling you everything you need to know about this.

How DataHub’s unified ingestion for Redshift works

Subscriptions and Notifications

One of our most requested features was the ability to subscribe to entities and get notifications about changes.

As we dived deep into this with community interactions and surveys, we saw the following themes/requirements emerge

• Getting ahead of breaking changes (breaking changes may come from unknown upstream entities)
• Understanding how new governance classifications drive compliance
• Optimizing for self-service
• Notifying data consumers on Slack and Email.

Here’s how this exciting project is shaping up:

Brittanie, with a sneak peek into the Subscriptions and Notifications features

Better Together: Community Contributions and Shoutouts

A massive shout-out to the wonderful humans working behind the scenes to keep the DataHub project thriving!

Check out this month’s entry by community member Venkata Krishnan: Starting a Data Governance Journey.

Inspired by our conversations with the Community, we’ve compiled some tips on rolling out DataHub within your organization. Here’s Paul sharing 5 Tips for Rolling out a Data Catalog from the DataHub Community.

Paul shares 5 tried and trusted tips for rolling out DataHub to the rest of your organization.

Paul shares 5 tried and trusted tips for rolling out DataHub to the rest of your organization

That’s it from me for now, folks!

See you in Slack 🙂