DataHub Community Updates: The Feb’23 Low-Down
Hello, DataHub Enthusiasts!
It’s March already (how?!), which means it’s time to share what the DataHub Community has been up to in February.
There’s a LOT that the fantastic Community has done to make DataHub better, easier to use, and all-around amazing.
Let’s get right to it.
Growing Strong: DataHub Community Updates
Our Slack Community now has 6400+ outstanding members, with over 400 folks joining us in February. We continue to have nearly 1,000 active users every week, and our emoji game is still going strong (up 200+ from last month! :D)
If you’re new on your DataHub journey, remember to join our Slack ! The Community is incredibly supportive, kind, and eager to help 😊
Upcoming Events in March
DataHub<>BigQuery Workshop
For those just getting with DataHub, we’re hosting an interactive session on BigQuery ingestion. We have limited seats, and it’s happening this Thursday, so remember to RSVP here.
Meet us at Data Council in Austin, March 28–30
The Core DataHub Team is stoked to head to Austin this month for Data Council. We’re packing a ton of content into these three days. Mark your calendar for the following:
- Shirshanka Das’s keynote talk on March 28th: Building a Control Plane for Data
- Paul Logan and I are running a hands-on workshop to help folks get up & running with DataHub on March 28th: URGENT! Help these Pets Find Homes: Working Across Teams in DataHub
- I’m speaking with Kyle Eaton from Great Expectations on March 29th: Teamwork Make the (Open Source) Dream Work: The Power of Cross-Community Collaboration
Lastly, Acryl Data, and Astronomer are excited to invite you to an evening of tacos, beverages, and fun during Data Council. Join us at Lazarus Brewing on Wednesday, March 29th, at 5:30 pm for a night filled with networking opportunities, delicious food, and great company.
RSVP here, and we will send Uber codes to all attendees for transportation to the event.
Pssst: If you still need to sign up for Data Council, use the code DataHub20 to get 20% off your ticket.
Community Case Study: Hurb’s Journey with DataHub
We heard from Patrick Braz & the Hurb Team during the February Town Hall about their DataHub adoption journey. Patrick shared how they use DataHub for easier discovery, security, and better decision-making, with fast-growing assets, new solutions, and growing complexity.
Check it out here:
DataHub + Herb = Data Hurb?
What’s Happening: Roadmap Updates
We launched v0.10.0 in early February, a massive release for us with many additions and improvements. Here’s a quick snapshot of what we shipped:
I’m most excited about
- Time-aware lineage, which is SUCH a great addition to an already amazing feature.
- Schema coverage: We’ve added support for JSON schema in this release, and we’ll continue to improve our coverage of all kinds of schemas in the ecosystem.
On the roadmap front, here’s how we’re doing. Things are in motion, and I can’t wait to see how they land 🙂
Progress toward our Q1’23 Roadmap as of Feb’23
Product Updates: Improving and Simplifying DataHub
Improvements to DataHub Search
We’ve improved DataHub’s search experience to implement a scrolling search. We moved on from using an Elastic Search API approach that prevented you from searching past 10,000 search results. A bonus is that this also fixes our long-standing CSV download issue for large search results.
This update makes our Search feature so much more effective, intuitive, and valuable, especially when searching across lineage — where there’s much more complexity within the metadata graph.
Ryan Holstien shows off DataHub’s new and improved Search experience
Redesign of the Queries Tab
You asked, we delivered. The Community needed a way to view, document, and share other commonly used SQL queries (say, those not executed by a daily job but run by analysts in an ad hoc fashion). We needed a curated query set — alongside top or recently executed queries.
Our redesigned Queries tab on the dataset page helps you
- Discover more queries
- Add an element of human curation, and thus
- Pack in more context for better discovery.
John shares how DataHub redesigned Queries tab can help with better discovery and context
Unified Redshift Ingestion
Previously, our Redshift ingestion source required configuring two connectors: one for metadata & lineage ingestion and another to extract top queries & usage. We have unified the Redshift ingestion — and brought along a few more changes you’ll like.
Here’s Tamás telling you everything you need to know about this.
How DataHub’s unified ingestion for Redshift works
Subscriptions and Notifications
One of our most requested features was the ability to subscribe to entities and get notifications about changes.
As we dived deep into this with community interactions and surveys, we saw the below themes/requirements emerge
- Getting ahead of breaking changes (breaking changes may come from unknown upstream entities)
- Understanding how new governance classifications drive compliance
- Optimizing for self-service
- Notifying data consumers on Slack and Email.
Here’s how this exciting project is shaping up:
Brittanie, with a sneak peek into the Subscriptions and Notifications features
Better Together: Community Contributions and Shoutouts
A massive shout out to the wonderful humans working behind the scenes to keep the DataHub project thriving!
You’re invited to contribute to the DataHub Community blog!
If you have something to share with the Community — about data governance, exciting data discovery projects, or DataHub deployments — contribute to the DataHub Blog Community Program.
Check out this month’s entry by community member Venkata Krishnan: Starting a Data Governance Journey.
Inspired by our conversations with the Community, we’ve compiled some tips on rolling out DataHub within your organization. Here’s Paul sharing 5 Tips for Rolling out a Data Catalog from the DataHub Community.
Paul shares 5 tried and trusted tips for rolling out DataHub to the rest of your organization.
Paul shares 5 tried and trusted tips for rolling out DataHub to the rest of your organization
That’s it from me for now, folks!
See you in Slack 🙂