MYOB Balances the Books on Data Reliability

Note: This story was originally published July 2023.

Challenge

MYOB is a cloud-based and desktop accounting software and business management solutions provider for small and medium-sized businesses in Australia and New Zealand. Their data platform journey began in 2020 with the implementation of a data mesh architecture using Snowflake and dbt at its core. As the platform matured and more internal and third-party data producers and consumers joined, the complexity of data dependencies grew.

With more than a thousand dbt transformations creating intricate dependency trees, schema changes to upstream datasets frequently cascaded into breaking changes downstream.

The core challenge was visibility and communication. Data consumers would discover that their datasets, reports, or dashboards were broken without any advance warning when upstream dependencies changed. This reactive approach created frustration across teams and undermined confidence in the data platform’s reliability.

Solution

To reduce the frequency of breaking changes and improve coordination between data producers and consumers, MYOB turned to DataHub Cloud as the foundation for a more transparent, proactive metadata workflow.

With DataHub, owners of downstream datasets can now use the UI to explore lineage and identify upstream dependencies. Then, reach out directly to those upstream owners through Slack or other channels to coordinate dbt transformation updates. This new visibility was a critical milestone in enabling collaborative schema change management.

To take it further, MYOB embraced a shift-left approach by embedding automated schema-change notifications directly into their CI/CD pipelines. All dbt transformations live in GitHub and are executed via Buildkite pipelines. A pivotal new step, “Notify data consumers”, was added as the final stage of each pipeline. This step prevents a Pull Request (PR) from being merged until it’s manually unblocked and downstream data consumers have been alerted.

The notification step runs a containerized script that:

• Detects dbt file changes in GitHub PRs (updates, deletions, or renames with content changes)
• Uses DataHub’s API to identify first-level consumers of affected tables
• Automatically generates and sends notification emails to dataset owners
• Publishes messages to a notification queue for broader organizational integration

This automation ensures data consumers are notified before changes are merged to the default branch; empowering them to review, comment, and collaborate on updates early in the development cycle.

Impact

With DataHub, MYOB transformed their change management process and realized big improvements across reliability, productivity, and collaboration.

Key outcomes included:

• Nearly eliminated breaking changes, reducing incidents from multiple per week to virtually zero despite 4x growth in Snowflake usage
• Automated change notification workflow ensuring all downstream dataset owners receive advance warning before schema modifications reach production
• Improved platform reliability by implementing shift-left data quality practices that proactively catch potential issues
• Transformed collaboration culture between data producers and consumers
• Enhanced team productivity by replacing reactive firefighting with proactive collaboration, allowing teams to focus on value-creation rather than incident resolution