Future-proofing your data
![](https://images.ctfassets.net/1u65qlkmktuw/2pY4hLQ4JKwmHqEyEO8jH0/628447fbc6988b3e4c8f70a275491c86/Blog_WYS_Tal.png)
Welcome to data nirvana! Zip ANZ's Director Data & Analytics Tal Bergman shares learnings from the transformation of our data stack and explains what it means for the team to be able to work with fast and future-proof data.
Imagine this! You’ve built a single data platform that powers your entire business. All workloads from your legacy data platform have been migrated and the old platform has been turned off. You breathe a sigh of relief knowing that 100% of the team’s focus is dedicated to the new world, the new platform. It is really fast to execute use cases today across analytics and AI, but how do you ensure the data platform remains fast for the next 12 months, 2 years, 3 years and beyond. How do you ensure all your efforts are not wasted and that complacency doesn’t set in?
The data strategy I set two years ago at Zip was to build fast and future-proof data. The journey to data enlightenment wasn’t straightforward. Zip’s original data platform built during the start up phase was slow, cumbersome, and riddled with duplications. It was exhausting and painful to work with.
A data analyst who is living the pain of legacy, duplicate data everywhere. Everything to do with data is slow, difficult and exhausting. This image was created with the assistance of DALL·E 3.
The transformation required building a modern data stack from the ground up, migrating hundreds of pipelines and tables, and ultimately switching off the old platform, something many companies struggle to achieve. The results were nothing short of miraculous, with data operations becoming fast, easy, and efficient. The team continues to blow me away with how quickly they can move on analytics use cases. Welcome to data nirvana!
A data analyst in data nirvana. Everything is fast, easy and relaxing. This image was created with the assistance of DALL·E 3
Looking towards the future, Zip is not resting on its laurels. The focus has shifted towards future-proofing our data ecosystem to ensure speed, efficiency, and governance remain in harmony. We spent a lot of time discussing as a team:
- How to keep things as fast as they are today for years to come?
- If we were to have a completely new data and analytics team every three months, could they be operational within a week?
A number of key themes emerged that we felt were critical to future-proof our data. These were that:
- Users need the ability to find, understand and trust data;
- Return on investment from data needs to be known; and
- Data changes must be delivered in a way where speed and data governance are in harmony.
There are many different ways to define data products, from a collection of data sets to fully defined and mature products. At Zip we define a data product as…
A product who’s primary goal is to drive business value using data
We define them in this way as it anchors individual products directly to value and enables us to provide a framework for delivering that value successfully. We think about the data product as a collection of data and services we use to organise it into something that create values for internal stakeholders or directly for our customers or merchants.
For example, we personalise the shop feed in the Zip app so each customer has a unique set of merchant recommendations. The data and services we use to transform and store it, and the AI services we use to run the recommendation engine all combine to form the data product that we connect into our shopping feed.
Introducing the data product that rules all other data products
Coming back to the themes that we felt are critical to measure how we are progressing with future-proofing our data. We needed a way to determine whether any change that was made to the ecosystem of data and data products, by one of many data and analytics team members, improved the health of the ecosystem or degraded the health of the ecosystem based on a set of key information guided by the abovementioned themes.
A slice of the ecosystem depicted via dbt lineage of data connected to data products.
I have to give credit to Moss Pauly, our Senior Manager — Data Products who came up with a elegant and simple solution to:
- Capture information from every change to the ecosystem
- Store the output and enhance with data product information through automation and a UI for manual entries
- Use the data product that rules over all other data products to drive behaviours across the data and analytics community to uplift health of the ecosystem and to future-proof our data.
Credit also goes to Leroy Kahn, our Data Management Lead, who brought a wealth of knowledge into the process on things that were important to measure.
As Zip continues to innovate, our journey from data chaos to nirvana is just the beginning. It is critical that we remain as fast as we are today for years to come. We have plans to leverage generative AI to speed up our documentation process on all data changes, ensure we have data quality tests across all critical pipelines and support authors and approvers of code with an AI oracle to lift the bar on coding standards and practices so that we continue to future-proof data at Zip.
To close I want to share several key learnings that may seem straightforward however were more significant than expected now that these learnings are in the rear view mirror.
- You can’t move forward if you are weighed down by the platform of the past. Migrate and decommission the legacy. Get it done!
- Optimisation has equal standing with building new things. As you build new take the time to optimise. The longer term benefits are there. We amaze ourselves on what we can do but it is only possible through the optimisation work we delivered on the way.
- Set the culture and standards early on. Overcommunicate and spend the time to take team members on the journey. Course correct behaviours often with clear explanation of why future-proofing data is just as important, if not more, than building things quickly.
Like what you've read?
Check out these other articles
![](https://images.ctfassets.net/1u65qlkmktuw/3hx3TJwYmFdvWgohWVhqbw/802c5a60d00bc6fd5e82b1001a980982/Blog_tile.png)
August 27, 2024
Passionate and Proud in Purple
Author: Tara Pollock
![](https://images.ctfassets.net/1u65qlkmktuw/NYaFxZAt5fQ9ovnYtfMnk/5f46d27fa809d8721120a5df77cb65ef/Blog_WYS_Supreet.png)
July 22, 2024
This is my Zipstory: Q&A with Supreet
Author: Supreet Rumana
![](https://images.ctfassets.net/1u65qlkmktuw/4PO3XB13kIigJiNtYTSQje/55030e8a05abefd71a328b21fe414809/Blog_WYS_Viv_3.png)
June 12, 2024
Write your story with Pride - Q&A with Viv
Author: Viv Luu
![](https://images.ctfassets.net/1u65qlkmktuw/43NGTyE0xv3RRyy8YN8v5C/9e2ffef489fc3a433e4f4907d285d8c1/Blog_WYS_Mayuri.png)
June 11, 2024
This is my Zipstory: Q&A with Mayuri
Author: Mayuri Mukherjee
![](https://images.ctfassets.net/1u65qlkmktuw/3iel5OtRPhqgTlIye4Bqfg/f045e3be7ebcd53d42d5e814615b64f0/Blog_Pride_Ben_v2.png)
June 7, 2024
Write your story with Pride - Q&A with Ben
Author: Ben Maliel
![](https://images.ctfassets.net/1u65qlkmktuw/5UrxHJsDlKIF66TeoO4bc3/4d40c12a849b98382837c9923c363b73/Blog_Life_in_mobile_engineering_v2.png)
June 6, 2024
Life in the ANZ Mobile Engineering team
Author: Multiple contributors
![](https://images.ctfassets.net/1u65qlkmktuw/6YAvIZbsxSrFEosthofPvf/9681764abf41d5750c6c7cf284f36c9d/Blog_WYS_Jacob.png)
April 16, 2024
This is my Zipstory: Q&A with Jacob
Author: Jacob Turjman
![](https://images.ctfassets.net/1u65qlkmktuw/28XvCqx0DpZVBoyic66mwI/966df58494b2c2e2ed1ef36988777ede/Blog_video_Risk_US_v2.png)
April 10, 2024
Life in the US Risk team
Author: Priyamvada Kamra
![](https://images.ctfassets.net/1u65qlkmktuw/5UfEF3P6wIKm9vJV70tc9E/5615eed2f2f4bf3cfa79c996ef7050dc/Blog_video_Data_ANZ_v2.png)
April 9, 2024
Life in the ANZ Data & Analytics team
Author: Multiple contributors
![](https://images.ctfassets.net/1u65qlkmktuw/4BzRR1v3EsiARhOWascUq7/95d685898e0284256d4a9c4ce4cd450e/Blog_video_Product_US_v2.png)
April 9, 2024
Life in the US Product team
Author: Multiple contributors
![](https://images.ctfassets.net/1u65qlkmktuw/6TBsLbphzA2aOjcuRGsS0I/b49203c00450945b84f6536bfea95b25/Blog_video_Product_ANZ_v2.png)
April 9, 2024
Life in the ANZ Product team
Author: Multiple contributors
![](https://images.ctfassets.net/1u65qlkmktuw/5hXfKrFhKxbfIgnrjLlJFV/ab422f1c9e434b300c8a38c0fe5f2d7b/Blog_video_Tech_US_v2.png)
April 8, 2024
Life in the US Technology team
Author: Multiple contributors
![](https://images.ctfassets.net/1u65qlkmktuw/2aWXIEkP0M4Y2FJoXSsfW6/1708dacb3efcdfdb9b864ad72359bc8a/Blog_video_Tech_ANZ_v2.png)
April 8, 2024
Life in the ANZ Technology team
Author: Multiple contributors
![](https://images.ctfassets.net/1u65qlkmktuw/7n5S64JpqytRc04Ozv48az/627dd12f77579b9a85c0fafd396b010b/Blog_WYS_Ella_4.png)
March 24, 2024
This is my Zipstory: Q&A with Ella
Author: Ella Werman
![](https://images.ctfassets.net/1u65qlkmktuw/1v8NTZavBfb1rlbYqkZive/242cc80fba34488802fb34ef3e75f18d/Mardi_Gras_24.jpg)
March 5, 2024
Zipsters celebrate Mardi Gras in Sydney
Author: Multiple contributors
![](https://images.ctfassets.net/1u65qlkmktuw/48FLDF9kqS4VFzG20VksM7/51787511ce55cb585969760f7d00532b/Mentoring_tile_v2.jpg)
January 25, 2024
Mentoring at Zip - Q&A with Tim and Josh
Author: Multiple contributors
![](https://images.ctfassets.net/1u65qlkmktuw/4ChhuVmevIeVaz3vZ1snNz/e4f5885a15fbfe3a2ba304232108bd0a/ZipStar_US.jpg)
January 10, 2024
Meet our 2023 US ZipSTAR Award winners
Author: Multiple contributors
![](https://images.ctfassets.net/1u65qlkmktuw/1Y3347DVH4HBcyJCn3yfqe/3e9f737e7b0773fdbbd2f07419fefd4d/ZipStar_ANZ.jpg)
January 10, 2024
Meet our 2023 ANZ ZipSTAR Award winners
Author: Multiple contributors
![](https://images.ctfassets.net/1u65qlkmktuw/1IOWa46SaoTJH5BvJ34hDn/d68536c1b9e6c9b5097d4fd10e5272a7/Blog_Nitin.png)
December 22, 2023
Building a culture that’s Stronger Together
Author: Nitin Prasad
![](https://images.ctfassets.net/1u65qlkmktuw/4whK9a6Hc5fuJHFYp9hJKk/14af429f1920303fc8e7e7dbcb69430c/Prod_Design_blog_3.1.jpg)
December 20, 2023
Product Design UnZipped: The work we do, and the way we grow
Author: Mel Hambarsoomian
![](https://images.ctfassets.net/1u65qlkmktuw/2moeniXGE7I1psqEJqYxxI/800ce89f7cedf672fc8c7c726babd72d/Prod_Design_blog_3__1_.jpg)
December 10, 2023
Get to know Zip ANZ’s Product Design and Research team
Author: Multiple contributors
![](https://images.ctfassets.net/1u65qlkmktuw/3HuWShjs73sYpEebPBQYyo/bca38f974ba113e6c92bd21aa1d1f615/Blog_Sam_M.png)
December 7, 2023
Data UnZipped: Q&A with Sam
Author: Sam Manila
![](https://images.ctfassets.net/1u65qlkmktuw/6y6OwnUOQgbgw7kBXqYXLu/104ac92c0e6bee915015dc67b65416ee/Blog_Lionel.png)
November 16, 2023
Writing my own story: From employee number 5, to Zip’s Delivery Manager for New Zealand
Author: Lionel Yip
![](https://images.ctfassets.net/1u65qlkmktuw/4CD0eLEboeGdDIc1Hbt1GU/6265957dfc0e82035d5a9f2917d8d009/Blog_Alvin__1_.png)
November 9, 2023
Data UnZipped: Q&A with Alvin
Author: Alvin Rusly
![](https://images.ctfassets.net/1u65qlkmktuw/7GU9Qhk9Nw5Du6qjtPkmRp/f73060bac9b99d38238fed843d7a98bd/Blog_Mel_2.3.png)
November 6, 2023
Levelling up with Zip’s Product Design and Research team
Author: Mel Hambarsoomian
![](https://images.ctfassets.net/1u65qlkmktuw/4VPeHdIINnJfRskAwktfl3/7939d49023537ccba9006a70ff83965b/Blog_Archana_v2.png)
October 17, 2023
Engineering UnZipped: Q&A with Archana
Author: Archana Kanakamedala
![](https://images.ctfassets.net/1u65qlkmktuw/6rynJtBoeyWs1UEFQZaAfV/b3cd72c1c4d3f432028e1fb3fed3a980/Blog_Laure.png)
October 13, 2023
Making the most of my Volunteering Leave with the Snowy Brumbies
Author: Laure Salou
![](https://images.ctfassets.net/1u65qlkmktuw/3jzqN6HVJeJQ2kFWIpWfcs/ac557425bb529815537416ad5451d578/Blog_Hannah_v2.png)
October 5, 2023
My role building and growing a team that’s Stronger Together
Author: Hannah Scott
![](https://images.ctfassets.net/1u65qlkmktuw/14ybl7kd6jTDlrWs4COqde/928f34f19a133218e7fac742b1851f59/Blog_Zach_2.png)
September 21, 2023
Celebrating Wear it Purple Day
Author: Zach Rennick
![](https://images.ctfassets.net/1u65qlkmktuw/6QBQA6vu1dMf1pqzpaUu1i/2efad9ab800fde58c65b4d4bb03490ef/Zip_YCA.jpg)
August 23, 2023
Zipsters support the Future of Finance
Author: Lucy Lindsay and Jimmy Kelly
![](https://images.ctfassets.net/1u65qlkmktuw/6zYQTvdXOlkdqfb21coIhB/4ff01d93c974ea7790ae04dd58234cea/Blog_Priya_final__2_.png)
June 6, 2023
Risk UnZipped: Q&A with Priyamvada
Author: Priyamvada Kamra
![](https://images.ctfassets.net/1u65qlkmktuw/6XR9KWnLHYOmvrStdmwkNW/4a62d3e7717116621cae44d411c719f5/Zipsterversary_blog.png)
May 2, 2023
Celebrating our Zipsterversaries: February - April
Author: Multiple contributors
![](https://images.ctfassets.net/1u65qlkmktuw/4foJVHrzwzuhqxwQMafhTq/7e1536210b3caceaa8518314d0eb2ab9/Zip_DV_Collective.jpg)
March 21, 2023
Zipping it forward - our partnership with The DV Collective
Author: Anna Wei and Ying Zhang
![](https://images.ctfassets.net/1u65qlkmktuw/6xIhVnEp7KWBSYBJV4YkXg/7468638fcd3c8a007a1705d4ed964f93/IWD.jpg)
March 16, 2023
How we celebrated IWD 2023
Author: Multiple contributors
![](https://images.ctfassets.net/1u65qlkmktuw/3AYPAeXVHdh7WTqiqx9Kpv/6f0810bb166610072261c5871b0288c1/Blog_Mossy_final_2.png)
February 16, 2023
Building a fit-for-purpose modern data stack, part 2.
Author: Moss Pauly
![](https://images.ctfassets.net/1u65qlkmktuw/2q9KQBcn6M9GCnR7dZvbQH/4ef603cef069505886f1a0c315f264aa/Blog_Mossy_final_1.png)
February 13, 2023
Building a fit-for-purpose modern data stack, part 1.
Author: Moss Pauly
![](https://images.ctfassets.net/1u65qlkmktuw/34Hlee1tQ3q3B65klNvRBb/9d40534f4f1b500f49e870cb5b717faa/Blog_Mo_final.png)
February 15, 2023
Risk UnZipped: Q&A with Mohamed
Author: Mohamed Afifi
![](https://images.ctfassets.net/1u65qlkmktuw/3BgkzrIYL4ALqLU7pxmJJ/0877c66ded07c5a5eeb45c39dfbba87d/Blog_Mel__1_.png)
February 9, 2023
All Things Product: Mel Hambarsoomian
Author: Mel Hambarsoomian
![](https://images.ctfassets.net/1u65qlkmktuw/6XR9KWnLHYOmvrStdmwkNW/4a62d3e7717116621cae44d411c719f5/Zipsterversary_blog.png)
January 29, 2023
Celebrating our Zipsterversaries: November - January
Author: Multiple contributors
![](https://images.ctfassets.net/1u65qlkmktuw/27rGuPkt1KJSZQAOBFnmb3/cf6ed703ca92bc6a21e7758906e6967d/LNY_blog__2_.png)
January 26, 2023
How our Zipsters celebrated Lunar New Year
Author: Multiple contributors
![](https://images.ctfassets.net/1u65qlkmktuw/aL61A9dV1eO61KecBFxWp/fb6e11586ead818b6103ed827ca45f2e/Blog_ZIF.png)
January 9, 2023
Making use of our Volunteering Leave
Author: Multiple contributors
![](https://images.ctfassets.net/1u65qlkmktuw/PD6N6gsPR6cycprZCrQHk/d604fb550aeb3c4386296dd8ab82e047/Blog_Autumn_final_2.png)
November 15, 2022
Engineering Unzipped: Q&A with Autumn
Author: Autumn Ragland
![](https://images.ctfassets.net/1u65qlkmktuw/7GU5yPGZgczLissr8fo9fI/9b9002fd21d3214f595b50f739677755/Blog_GG_final.png)
November 11, 2022
Atomic change and the future of Technology at Zip
Author: George Gorman
![](https://images.ctfassets.net/1u65qlkmktuw/7FUIu4i3NWLgjx9l31beFr/64991eecf48b539ed49d7efc6460f3d5/Blog_Tal_final.png)
November 11, 2022
Moving at Zip speed while building data that stands the test of time
Author: Tal Bergman
![](https://images.ctfassets.net/1u65qlkmktuw/6HnTSYMB0kPPcUM5FY32MI/c1940cc9b865189a266b8083ae2b050e/Blog_Kalpana_final.png)
November 11, 2022
Engineering UnZipped: Q&A with Kalpana
Author: Kalpana Chandrasekar
![](https://images.ctfassets.net/1u65qlkmktuw/7u3z6Xhgt5V0oeQqWQTwoW/5fc756dd0a51c215a64d21133691d06e/Blog_WW_final.png)
November 11, 2022
Master these three skills to grow your data and analytics career
Author: Will Walker