All articles

Building a fit-for-purpose modern data stack, part 2.

Author:

Moss Pauly

Published:

February 16, 2023

Building a fit-for-purpose modern data stack, part 2.

This is the second piece of a multi-part blog post by Sydney-based Zipster Moss Pauly, Senior Manager, Data Products, detailing the creation of our ANZ data platform and the team’s journey over the last 18 months.

Part one set the scene for how the team established a framework to guide decision making on the various components that now make up Zip’s stack.

In this final, longer instalment, Moss deep dives into a summary of the key decisions the team tackled when modernising Zip’s data stack, and excitingly what comes next. So grab a coffee or drink of your choice, get comfortable and dig in….

Thanks for stopping by again. As I mentioned in the first part, we didn’t want to make decisions on components in this stack lightly. Here I share an in depth reflection of six key decisions we made regarding the stack and its component parts - hopefully you find the takeaways and tips useful as part of your own journey.

Decision #1 — How are we going to transform our data?

We started here as we wanted to nail how we store business logic and transforms to build data models in a scalable, future-proof way. We didn’t want this decision bounded by constraints on our data warehouse, ingress, or egress compatibility.

We had the following core requirements:

  • The tooling needed to be able to support DAG type dependencies (n) layers deep.
  • It needed to be SQL-based.
  • We needed ways of tying documentation (e.g. metadata) directly to models in a way that we could check that they stayed in sync as part of our PR process.
  • Capitalising on existing models for third-party data sources would be nice to have.

Dependency management and documentation were both significant pain points of our existing transformation stack which in essence was Airflow orchestration of SQL, through Athena, on an S3 Data Lake. While we looked at a few options in this space, dbt was the clear winner here. The sentiment around it is fantastic and this became very clear quickly in conversations we had with other dbt customers.

With the framework we mentioned above, we went through a POC before documenting the outcome and recommendation to proceed with dbt. We then formalised a decision with our stakeholders.

After 18 months and with 1000+ models in production, we’re pretty happy with how dbt’s going. We’re currently on the cloud instance and the new hosted IDE is a game changer. We hit scaling issues pretty badly with our project size initially, but these are mostly resolved now.

Top tips for setting yourself up for success with dbt:

  • Enforce documentation and test requirements from the start with dbt_meta_testing.
  • Read gitlab’s dbt guidebook, it’s an incredible resource.
  • Spend the time to go deep on understanding macros and incremental models. They’re both insanely powerful.

Decision #2 — Data storage and compute

The second component that we looked at was the data warehouse itself — the compute and storage that dbt runs on. Given that we have a data science community operating in Databricks, we were keen on evaluating Delta Lake as an architecture alongside Snowflake as a data warehouse. We did extensive POCs on both, as we knew that this was not a decision we’d reverse easily and that it’d have an enormous impact on the quality of life of all data practitioners at Zip over the coming years.

There have been some entertaining fights in the data space of late about the performance of these solutions. As general advice, I’d say performance should be at the bottom of your list when it comes to making a decision. The reality is they’re all fast, who really cares about the basis points of speed in comparison to the quality of life working with it day to day?

The main components we evaluated here were:

  • If you wanted to run a quick query, what’s the time to result? (Opening the tooling, navigating it if required, waiting for a cluster spin up etc…)
  • What granularity of cost visibility can we have easily?
  • How well written is the documentation and how easy is it to find answers to questions?
  • How well does it integrate with dbt?
  • What would the management impost be on the small team responsible for operating and maintaining the platform?
  • How easy will it be to hire people with previous experience on this or bring people up to speed from scratch on this?
  • How easily can we manage PII redaction in this stack to protect our customers’ privacy?

After evaluating all the above we settled on Snowflake being the right choice for us. One of the bits of advice I got from talking with someone in a company who switched about a year before us was There’s so much that really just works in it. Looking back it was close at the time but we’re really glad we made the decision that we did.

After using Snowflake for 18 months, I really couldn’t agree more with this. This is echoed throughout the data community at Zip.

See below for our high-level data warehouse architecture:

MP_data chart

Top tips for setting yourself up for success with Snowflake:

  • Spend the time to really consider your security model and work through how it can be applied with role inheritance.
  • If you went with dbt as well as Snowflake, look at and leverage dbt_snow_mask to dynamically mask all PII from the start.
  • Zero copy cloning is amazing for enabling you to test model changes on a production clone. Use and abuse it.

Decision #3 — Event Collection

Our previous stack's inability to deal with the volume of raw events coming in was one of the catalysts for kicking off our journey in the first place. We decided to look in detail at Segment, Rudderstack, and Snowplow. My partner in crime Adam Schmidt (Principal Engineer at Zip) and I had very strong opinions about what we needed in an event collection platform having both worked in this space relatively extensively in the past.

  • We did not want a solution where we’d constantly need discussions about what we can and can’t track due to cost.
  • We must be able to enforce schema compliance on events.
  • The solution must be scalable to billions of events a month and must be able to handle sudden spikes in volume gracefully.
  • We wanted to have first-party ownership of our events.

We decided to do a POC for Snowplow due to a strong preference for open-source solutions in this space, and Adam deployed it to production. After extensive evaluation, we decided that we were happy to take the overhead for managing this infrastructure ourselves in favour of not having a SaaS cost tied to Events/Month. This is the first decision I’ll outright say is not for everyone:

  • Snowplow is incredibly flexible, but that’s a blessing and a curse. There’s a bit of a learning curve in understanding how to configure and optimise the pipeline, and you need to have well-supported streaming infrastructure (in our case Kafka) at your disposal.
  • Our platform and tooling are mature, and we have a reasonably sized engineering team. If you’re a small-ish organisation, start with the cloud offering.
  • We’ve had to find some creative ways to provide tooling for developing/testing/troubleshooting event implementations
  • In addition to deploying the pipeline, there are a few somewhat special things about the way we integrated Snowplow which I’d strongly recommend implementing, regardless of how you collect events.

Top tips for instrumenting any event collection framework:

  • Traceability on every event collected to a specific system and team. We do this in Snowplow leveraging app_ids to each service. Any events that come through with an app_id not registered in our system get rejected. Additionally, every event is enriched with a service_name and team_owner based on a lookup on the app_id. This means that we can trace every single event in our platform to a team owner. Even though we have 20+ services sending us page_view events, we know exactly which team own each and every one that comes through.
  • Abstract the integration to enable centralised standardisation. We built out zip_analytics.js which is an opinionated abstraction library on the Snowplow SDK as well as a number of other analytics libraries. This makes our life so much easier than trying to coordinate 20+ teams to make a config change in their tracking implementation, and forces a predictable consistency in how event data arrives on the platform. This has already paid dividends with us rolling out a number of tweaks to how we track events over a very significant number of integrations with almost no coordination work needed.
  • Ensure that the data exiting the pipeline can be loaded into your warehouse with minimal fuss, especially in the early stages where you’re iterating and extending your event models frequently. The loader should be as dynamic as possible to reduce engineering effort in adding and modifying events/contexts.
  • Spend a bit of time up-front ironing out your principles for event namespaces, event names, and schema conventions. Use your most popular customer feature as a real-world test bed for how you intend to work with events going forward.

Decision #4–Third-Party Data Ingestion

We deliberately evaluated solutions for first-party and third-party data separately due to the following reasons:

  • Our appetite for an acceptable cost per data point is different between first-party and third-party data due to the massive differences in volume between these.
  • Data modelling can theoretically be a solved problem for a lot of third-party data sources, this is not the case for first-party data.
  • If you’ve been in the data space for a bit, you’ve likely had to deal with reverse engineering modelling the data out of a third party. Maybe this wasn’t too bad (shoutout to Twilio’s data structures ❤) or maybe you spent months of your life trying to model a Salesforce data feed (you have my sympathy). We were keen to capitalise on standard models in dbt for third-party data sets where possible.

Fivetran was active in this space, so we decided to do a POC with them. We evaluated a few other options on paper, but Fivetran came out ahead. This decision focussed on:

  • Coverage of our core third-party data feeds.
  • The shape of landed data structures.
  • Documentation of data structures.
  • Prepackaged dbt transform availability.

In all the above situations we were happy with how Fivetran performed during our POC.

Recently, our CIO was wanting to query Twilio data and pinged me about the Twilio table structure while I was getting coffee. I was able to send back a link to the Twilio ERD in Fivetran about 5 seconds later that fully explained everything. Well-documented third-party integrations are really valuable.

Top tips for Fivetran:

  • We pulled out some of the prepackaged dbt transforms from Fivetran that we initially had in for IDE performance reasons. I suspect this isn’t as much of a problem anymore with dbt’s new IDE though. Moving forward, our sensible default is “add them when you need them not when you can”.
  • Use Fivetran to send its own usage logs to your warehouse. It gives you a nice set of tables to monitor consumption and usage centrally in your BI tool of choice.

Decision #5–First-Party Data Ingestion

At Zip, we have a pretty significant amount of first-party data. This is highly distributed through a core monolithic SQL Server database, Kafka event streams, Postgres databases, and DynamoDB tables that back our microservice architecture. We needed a scalable way of dealing with the rapid ingestion of all these in an hourly fashion, and the cost of doing this volume of data in Fivetran was not feasible for us so we started looking elsewhere.

Similar to paying for event ingestion, we really wanted to avoid needing to have conversations about what we do and don’t ingest based on cost. We looked again at the open-source community here and discovered Airbyte, which was a relatively young open-source project that looked promising.

Our requirements were as follows:

  • Support SQL Server
  • Support of CDC on SQL Server
  • Support Postgres
  • Support S3 Ingestion
  • Snowflake Destination Support
  • Cost Efficient
  • Sub 1-hour integration of a new data source and/or tables
  • Connector-based with a rich collection of pre-canned connectors, and the ability to build our own if we wanted to.

Airbyte fitted these requirements pretty well, so we started our POC again deploying to our production environment. This journey was a little bumpier than some of our others, with one of our data engineers submitting over 10 PRs to the project remediating some issues with the MSSQL connector. The Airbyte community and core maintainers are really active and supportive, so getting the changes merged was not a problem.

That aside, we’re really happy with where we landed. Ingestion is quick, it’s stable, and it meets our needs. We’re doing hourly ingestion from a number of databases big and small and we haven’t needed to touch this much after the initial setup.

Decision #6–Data Egress

We started a conversation with Census in this space back in the early days when we were looking at Snowflake and dbt, although this is one of the most recent additions to the data stack.

We ended up waiting until we had a number of use cases we could solve with this tool to make the business case and get this across the line. We ended up doing a full POC on Grouparoo and Census. Grouparoo is an open-source project recently acquired by Airbyte, due to our existing experience with that project and our preference for open-source solutions it made sense to see if this would work for us. While Grouparoo initially looked promising, we decided we weren’t comfortable with its maturity. The issues here were primarily due to connection instability during scaling and the methodology for existing record checks in the Braze connector.

I can’t wait to see what Airbyte does with Grouparoo in the near future, but it didn’t fit our needs at the time of evaluation. A lot of our use cases for data egress revolve around shipping data into Braze, our CRM platform.

The key aspects we were evaluating are below:

  • Sub-3-hour updating of 4 million records in Braze
  • Robust updating of only records that have changed (Braze charges per data point updated so this is a must for cost efficiency)
  • Friendly UI to empower product and marketing teams to control what data ships where.

We were delighted with Census from the start and decided to move forward with them — a decision we’re really happy with. Since then, we have started working on some creative aspects with their segment builder empowering our product and marketing teams, which I’m sure we’ll share in a future blog post.

Top tips here:

  • Triggering Census syncs post-completion of dbt jobs is your friend for seamless integration.
  • Entity models have been released recently. These are game-changing when it comes to enabling non-technical people to create segments that would normally need to do complex joins across a large number of data sources.

What’s next?

The above forms the core of the data platform at Zip.

However, I view this as Building our data platform is really just the entry ticket into being a modern data-led company. We’re currently well underway with a host of next-phase activities built on a strong foundation:

  • Enabling our data science teams with Feast as a feature store
  • Visualising model performance using Streamlit applications
  • Evaluating data catalogue options to move data trust and data definitions from being predominately tribal knowledge to explicitly documented and easily accessible
  • Monitoring and alerting for data SLA breaches, data profiling and core funnel performance across 40k merchants.

I’m proud to have been part of the journey of modernising the data stack at Zip, and couldn’t have done it without the outstanding effort of the data engineering team (special shout out to Adam Schmidt, Kiran Reddy, Nick Heron, Thomas Gao, Alex Crawford, Frank Fernando and Sivakumar Ramaswamy), data analytics team, and data science teams. Additionally, this would not have been possible without the data leadership at Zip (Tal Bergman — Director of Data and Analytics and Adam Finger — CIO) recognising the opportunity in this space and backing our journey into it.

Like what you've read?

Check out these other articles

June 12, 2024

Diversity, Equity & Inclusion
Write your story with Pride - Q&A with Viv

Author: Viv Luu

For this next blog, Team Lead for Credit Lending Viv Luu reveals what attracted her to Zip four years ago and how she gets her best work done

June 11, 2024

Data, Analytics & Risk
This is my Zipstory: Q&A with Mayuri

Author: Mayuri Mukherjee

In this article, Mayuri, a Canada-based Zipster talks us through her Zipstory, and how her life experiences have shaped her passions for her work in the US Risk team

June 7, 2024

Diversity, Equity & Inclusion
Write your story with Pride - Q&A with Ben

Author: Ben Maliel

Meet ANZ Product Manager Ben to hear how he has adjusted to life at Zip, and what Pride Month means to him

June 6, 2024

Technology
Life in the ANZ Mobile Engineering team

Author: Multiple contributors

Welcome to the Zipster blog Jacky, Nilu and Eddie! Jump in to hear how their work in our mobile engineering team is changing the game for customers, and Zipsters' careers

May 1, 2024

Data, Analytics & Risk
Future-proofing your data

Author: Tal Bergman

Welcome to data nirvana! Zip ANZ's Director Data & Analytics Tal Bergman shares learnings from the transformation of our data stack and what it means to work with fast and future-proof data.

May 1, 2024

Commercial
This is my Zipstory: Q&A with Lucy

Author: Lucy Lindsay

Meet Lucy Lindsay from the ANZ Commercial team who reveals what has enabled her multi-role progression and what makes her a proud Zipster.

April 16, 2024

Technology
This is my Zipstory: Q&A with Jacob

Author: Jacob Turjman

In this blog, Jacob from the ANZ IT team tells his Zipstory, from being permanent IT team member number 1, to his current role leading six Zipsters.

April 10, 2024

Zipster video
Life in the US Risk team

Author: Priyamvada Kamra

[Video post] - Get to know Priyamvada, who has owned it from day one as a Zipster, maturing our US-based Risk strategies while maintaining a healthy work-life balance.

April 9, 2024

Zipster video
Life in the ANZ Data & Analytics team

Author: Multiple contributors

[Video post] - Introducing Niki and Will from team Data & Analytics. Watch to learn more about the game-changing work they have owned while writing their Zipstories.

April 9, 2024

Zipster video
Life in the US Product team

Author: Multiple contributors

[Video post] - Introducing Trygve and Emily, two Product Managers who are putting their name to no shortage of interesting initiatives to support the growth of the brand in the US.

April 9, 2024

Zipster video
Life in the ANZ Product team

Author: Multiple contributors

[Video post] - Hear from Narelle and Jay, two OG Zipsters who have shaped their careers in different and interesting directions in the ANZ Product team.

April 8, 2024

Zipster video
Life in the US Technology team

Author: Multiple contributors

[Video post] - Meet Kibi, Robert and Elijah from Zip's US Technology team. In this video, each Zipster reveals what differentiates working in Tech at Zip from their previous careers.

April 8, 2024

Zipster video
Life in the ANZ Technology team

Author: Multiple contributors

[Video post] - Hear from Dan and Fatima, two fearless Zipsters in the ANZ Technology team, to understand more about how they are owning the latest chapter of their Zipstory.
This is my Zipstory: Q&A with Ella

Author: Ella Werman

Meet Senior Merchant and Consumer Marketing Manager Ella Werman to hear all about how her work is driving incremental value for the company and our merchant partners.

March 5, 2024

Diversity, Equity & Inclusion
Zipsters celebrate Mardi Gras in Sydney

Author: Multiple contributors

As Sydney geared up for another spectacular Mardi Gras, we asked Zipsters from our Zip Pride community to share what the event means to them.

January 25, 2024

Life at Zip
Mentoring at Zip - Q&A with Tim and Josh

Author: Multiple contributors

Meet our mentor/mentee pair Tim Danusa and Josh Wilson, who share their experiences at the half way point of our inaugural Zip Mentoring Program

January 10, 2024

Life at Zip
Meet our 2023 US ZipSTAR Award winners

Author: Multiple contributors

Get to know the winners of our 2023 ZipSTAR Awards, recognising US-based Zipsters who have lived one or more our four values in all they've done

January 10, 2024

Life at Zip
Meet our 2023 ANZ ZipSTAR Award winners

Author: Multiple contributors

Get to know the winners of our 2023 ZipSTAR Awards, recognising Zipsters who have lived one or more our four values in all they've done
Building a culture that’s Stronger Together

Author: Nitin Prasad

We recently sat down with our Senior Product Manager Nitin Prasad, who’s based in our Takapuna (Auckland) office in New Zealand. Here's what he had to say about culture, communication and cross-functional work at Zip.
Product Design UnZipped: The work we do, and the way we grow

Author: Mel Hambarsoomian

In the final instalment of a 3-part series, we go behind the scenes with our ANZ Product Design team to understand more about the work they're proud of, and how they're growing.
Get to know Zip ANZ’s Product Design and Research team

Author: Multiple contributors

This is the second of a three-part blog series profiling our Australia and New Zealand Product Design and Design Research team

December 7, 2023

Data, Analytics & Risk
Data UnZipped: Q&A with Sam

Author: Sam Manila

Get to know what a typical day in the life looks like for Zip's Senior Manager, Data Science for ANZ Sam Manila.
Writing my own story: From employee number 5, to Zip’s Delivery Manager for New Zealand

Author: Lionel Yip

Based out of Auckland, New Zealand, Lionel Yip is a Delivery Manager in our Technology team. Read on to discover more about his journey with Zip.

November 9, 2023

Data, Analytics & Risk
Data UnZipped: Q&A with Alvin

Author: Alvin Rusly

Meet Alvin, a Sydney-based Senior Data Analyst, whose work is helping Zip's New Zealand team make better data-driven decisions
Levelling up with Zip’s Product Design and Research team

Author: Mel Hambarsoomian

In the first of a three part blog series profiling our ANZ Product Design team, hear from Director Mel Hambarsoomian about the team's focus on design maturity, tri-track agile and craft and quality.
Engineering UnZipped: Q&A with Archana

Author: Archana Kanakamedala

Say hi to Archana Kanakamedala, and learn who and what have inspired her journey and career progression to her current role of Engineering Manager in our ANZ Core Product Squad.
Making the most of my Volunteering Leave with the Snowy Brumbies

Author: Laure Salou

Meet Lead Product Designer for our ANZ business Laure Salou and learn how she discovered a new passion while Volunteering on Zip’s time.

October 5, 2023

Candidate Support
My role building and growing a team that’s Stronger Together

Author: Hannah Scott

Read on to hear how our ANZ Senior TA Partner Hannah Scott’s focus on candidate excellence has helped develop her passion for DEI, and get her top tips on how to excel in your recruitment process and your career as Zipster

September 21, 2023

Diversity, Equity & Inclusion
Celebrating Wear it Purple Day

Author: Zach Rennick

We recently sat down with CX Superstar Zach Rennick in celebration of Wear it Purple Day, an annual LGBTIQA+ awareness day especially for young people, based in Australia.
Zipsters support the Future of Finance

Author: Lucy Lindsay and Jimmy Kelly

Hear from Sydney-based Zipsters Lucy and Jimmy who volunteered their time to partner with Young Change Agents, a nationwide social enterprise helping youth from all backgrounds see problems as opportunities

June 6, 2023

Data, Analytics & Risk
Risk UnZipped: Q&A with Priyamvada

Author: Priyamvada Kamra

Meet Priyamvada, our Director, Risk Strategy based in the US. Read on for her career story, the work she's leading at Zip and how she brings her true, authentic self to work.

May 2, 2023

Life at Zip
Celebrating our Zipsterversaries: February - April

Author: Multiple contributors

Each month we recognise the tenure of a Zipster who has truly lived our values during their time here. This article re-caps the stories of the three Zipsters who we celebrated from February to April.
Zipping it forward - our partnership with The DV Collective

Author: Anna Wei and Ying Zhang

Meet two of our fearless Product Managers, Anna Wei and Ying Zhang, and learn how they used their Volunteering Leave to support The DV Collective, a national domestic violence initiative.

March 27, 2023

Diversity, Equity & Inclusion
Inspiring our next generation of female sales leaders

Author: Karen Farrar

In this article, Senior Director, Commercial Karen Farrar reflects on this year's International Women's Day theme and how it resonates given her experiences as a Zipster.

March 16, 2023

Diversity, Equity & Inclusion
How we celebrated IWD 2023

Author: Multiple contributors

Learn all about how we recognised International Women's Day 2023 and what a gender equal future looks in the eyes of four of our fearless Zipsters.
Sustainability UnZipped: Q&A with Oli

Author: Oli Nelson

We recently sat down with Zip’s Sustainability Associate Oli Nelson to hear what the team have been up to in support of our global Social Impact and Sustainability initiatives.

February 13, 2023

Data, Analytics & Risk
Building a fit-for-purpose modern data stack, part 1.

Author: Moss Pauly

In the first instalment of a multi-part blog post, Sydney-based Zipster Moss Pauly introduces our ANZ data platform and the team’s journey over the last 18 months.

February 15, 2023

Data, Analytics & Risk
Risk UnZipped: Q&A with Mohamed

Author: Mohamed Afifi

Meet Mohamed, based out of our NYC office, who's leading our Enterprise Risk Management strategy at an exciting point in the journey for Zip in the US.
All Things Product: Mel Hambarsoomian

Author: Mel Hambarsoomian

Our Director of Product Design, Melanie, recently sat down to reveal her career highlights, sharing what she looks for in talent, her insights on design maturity, product thinking and design leadership.

January 29, 2023

Life at Zip
Celebrating our Zipsterversaries: November - January

Author: Multiple contributors

Each month we recognise the tenure of a Zipster who has truly lived our values during their time here. This article re-caps the stories of the three Zipsters who we celebrated from November '22 to January.

January 26, 2023

Diversity, Equity & Inclusion
How our Zipsters celebrated Lunar New Year

Author: Multiple contributors

Hear about the significance of Lunar New Year to Jess and Clement, and learn how they celebrate.
Making use of our Volunteering Leave

Author: Multiple contributors

Every year Zipsters get 2 days of paid Volunteering Leave. Here's how a couple of the team used theirs...
Engineering Unzipped: Q&A with Autumn

Author: Autumn Ragland

US-based Engineer Autumn shares her experiences as a Zipster.
Atomic change and the future of Technology at Zip

Author: George Gorman

ANZ CTO George Gorman gives a glimpse into the opportunities that lie ahead for the team.

November 11, 2022

Data, Analytics & Risk
Moving at Zip speed while building data that stands the test of time

Author: Tal Bergman

ANZ Director Data Tal Bergman offers insight into life in the Data & Analytics team
Engineering UnZipped: Q&A with Kalpana

Author: Kalpana Chandrasekar

Meet ANZ Engineering Manager, Kalpana Chandrasekar, from our Shop & Rewards team.

November 11, 2022

Data, Analytics & Risk
Master these three skills to grow your data and analytics career

Author: Will Walker

Looking to grow your data and analytics career? ANZ Senior Manager, Analytics Will Walker shares his top tips.
View all articles