In an oft-memed scene from the cult film UHF, the goofy host of a TV variety show shouts at his guest: “You get to drink from the firehose!” When it comes to managing and governing Kafka data, data teams can easily identify with that guest. It’s as if they’re tasked, hopelessly, with sipping from a…
It’s the home stretch for your company’s quarterly financial reporting, just hours before 7 AM—when earnings are due to be released. That’s when an eagle-eyed analyst spots an anomaly in the revenue data: the `deferred_revenue` column is missing for the last few days of the quarter. This means your company’s earnings report probably understates future…
The five-alarm data quality infernos always seem to break out in the middle of the night. Like during your closing period—just hours before finance is expected to finalize quarterly figures. That’s when a sharp-eyed analyst notices `booked_revenue` for several regions is off by an order of magnitude. She suspects an error in the data pipelines…
“Why is the revenue dashboard missing last week’s data?” “Did something change about how we calculate sign ups?” “Are our numbers are going to be incorrect again this week?” Questions like these are how data teams find out about data quality fires. Instead of being first to know about data issues—data teams are last, often finding out via…
Online payments powerhouse PayPal had a pretty common problem. Its teams couldn’t find the data they needed to do their work, either because it was siloed across separate environments, wasn’t properly documented, or lacked lineage and other essential metadata. So Vaidehi Sridhar, product manager at PayPal, came up with a clever solution: a hackathon—but for…
PayPal Product Manager Vaidehi Sridhar probably wouldn’t call herself a mastermind, but she did execute on a genius idea to improve data quality across PayPal’s sprawling ecosystem. Even better, Sridhar and her team laid the groundwork for PayPal to federate data governance for its decentralized teams—as well as automate many types of tedious governance tasks. The genius…
Hear that sound? It’s the sound a turntable makes when its stylus jumps the groove and skids across a record. It’s a sign the party has suddenly and irreversibly come to an end. Something like this may be happening with AI, as the best-laid plans of teams building AI solutions crash up against the operational…
When organizations struggle to operationalize ML or AI solutions, the root causes are usually data-related. ML and AI teams can’t find the data they need to define use cases, engineer features, or train their models. When they can find it, they can’t always use it—because they don’t know what it is, where it came from,…
Picture This You are a risk analyst with a financial services organization. You’re part of a team that creates products and services for small- and medium-sized business (SMB) customers. But … your team’s efforts are constrained by poor-quality data. You struggle to identify and integrate the data you need to analyze SMB market segmentation and…
Picture this You’re a data engineer at a boutique e-commerce start-up. Your company sells luxury goods at steep discounts. One of your many responsibilities involves monitoring the flash_sale_purchase_events table in your start-up’s Snowflake data warehouse. Updates to columns in this table are supposed to reflect real-time participation by customers in the limited-time flash sales your company offers. It’s mid-December….
