The data space has grown rapidly in last few years. To put this progress
in context Mahdi has divided it into three waves. First being of ETL, OLAP,
and relational data warehouses. The second being the era of scalability
and now we have entered the third wave - the Modern Data Stack wave.
Join us in this interesting conversation where Mahdi talks about various
aspects of Modern data stack like ETL, CDC, data obvervability and much
more.

Senior Data Engineer

Insights for CTOs: Part 3 – Growing your business with modern data capabilities

S01 E02: The third wave of data technologies with Mahdi Karabiben

Hello, everyone. You're listening to another episode of the modern data show. And today we have Mahdi Karabiben joining us from the beautiful city of Paris. Mahdi is a senior data engineer at Zendesk who enjoys working with data and building scalable data platforms. Zendesk is a customer service software that was founded in Copenhagen, Denmark in 2007. And have since grown to over 5,400 employees serving 160,000 paid customer accounts in 160 countries, powering billions of conversations with hundreds of millions of customers over telephony, chat, and many more channels. At Zendesk, Mahdi is responsible for building data products and enhancing the existing data stack to help internal team access product data at scale, and he loves to contribute to open source. Recently Mahdi also authored a live project series on 'Building end-to-end batch data pipelines with spark' with Manning Publications. Mahdi started his career at Democracy International helping Tunician ministries work with electoral data, and then joined an Adtech firm, Numberly to revamp the data architecture of one of the core products followed by a stint at Credit Agricole and FactSet.

Welcome Mahdi, we are super happy to have you on our podcast and thank you for joining us.

Aayush Jain

Thank you Aayush, Thank you for having me. I'm very happy about being on the podcast and thanks again for the invite. And I'm looking forward to talk about data and about data stacks.

Mahdi Karabiben

Amazing. Amazing. So before jumping into anything specific Mahdi let's first start with your work with Democracy International, how did that even happen?

Yeah, I mean, it was around six years ago. I was still a student. And so basically Democracy International is an American NGO who mainly works with governments on trying to democratize access to data. So, you know, for data that comes from ministries and from official sources, usually it's not very easy to actually understand and comprehend for any citizen. And so Democracy International basically tries to democratize access to that via building interactive charts and dashboards that are easily accessible and trying to build UIs for open data platforms that allow people to easily understand those data sets. So, yeah they reached out when I was still a student because I already started being active in some open source projects. And they reached out saying that, Hey, we, we have this project with Tunisian government to actually try to build platforms and UIs for different open data sets. And that was mainly what I worked on trying to just revamp that type of data and trying to think of how can any citizen access that data in a very efficient manner and a very interactive and intuitive manner to get a full picture of the data, whether it's municipality, whether about how the government is doing about budgets and so on. So it was lot of fun and it was, yeah, my first dive into data and trying to explore different data sets, data quality issues and so on, and it was quite interesting. Yeah.

Awesome. Awesome. And we talked about this when we met in person last month, but tell us a little bit about your journey from Tunicia to now in Paris.

Yeah, it was I think it was quite interesting for me since I started working with Democracy International, it was clear that I want to work in data. Like I, enjoyed the work so much. I enjoyed working with the data, like whether it's modifying it, trying to build dashboards and so on. It was really interesting work for me.
And so for my end of studies internship, I wanted to work with as much data as possible and try to maybe work on other parts of the stack, whether it's the data warehouse, whether it's building data lakes, just trying to explore that whole ecosystem and an opportunity with Numberly was available. So Numberly is basically an AdTech company based in Paris. They work on basically most of the marketing campaigns that are done here in France, whether it's via phone SMS, or emails, or even publications or ads on websites. And for that, we use a lot of cookies data, and we use a lot of data from external sources. They have a large Hadoop cluster and the project was very interesting. It was about migrating from existing pipelines that were done with Hive into Spark and Airflow. So it was a lot of work to be done and also a lot of optimization and going into very interesting parts of how Spark actually executes queries and so on. So it was a very interesting project. It was, that led me to come to Paris. And since then I'm here I remained in Paris and I remained working with data.

Amazing. And you know let's go, let's talk a little bit about Zendesk now. So you know, help us understand, how is the data organization at Zendesk structured and how does data work really happens at Zendesk?

Yeah. So I think it's, I'd say it's the typical path of any tech oriented company that grew really quickly. So you start from a point where you have that period of very quick growth in which there are acquisitions, there are new organizations new ways of working with data and so on. And so the whole tech stack grows in different directions very fast, and it's just keeping track with it will be a very hard task. And it's the same for data. Zendesk is by nature a very data-oriented company, and it's very data-driven, all the decisions. Actually, I was surprised when I joined how data driven the company is, every opportunity to actually leverage data in decision-making is taken, there's no way of just having data stay or having data available and not leveraging it. So with that, we are eventually many data teams, but some of them are created organically, some via acquisitions and so on. The three main teams within Zendesk are first is Zendesk Explorer, which is basically a Zendesk product, that allows, Zendesk customers to, have analytics and reports on how, their teams are using Zendesk and how many tickets are created. So behind that there is dedicated data engineering team that works on offering those capabilities and data apps to the Zendesk customers. Then there is foundation, which is the engineering team that works on foundational data platform, which is mainly for product data. How product teams actually publish their data, how data is stored, how it's processed and then how it's made available to internal teams within an internal platform. And the last team, which is the team I'm part of, which is EDA -enterprise data and analytics. We basically work on building the curated data sets and working on the data warehouse of Zendesk to actually work with the internal stakeholders to offer them curated data sets and access to data, whether it's for business purposes, or even for product analytics and so on.
So we have many data domains that we own. So for example, take finance data. How is the company doing financially? You can take product data. How are users actually using the features that we release? How are users actually interacting with different features of Zendesk products that would help product teams better organize their roadmap and better think about the features that they are releasing. So, yeah, in total, currently we're working on six data domains within EDA, but we're very small part of big picture when it comes to data at Zendesk that's explore foundation and then EDA, I think

So we saw a similar org structure when we spoke with Canva, you know, they have a very similar, federated, a structure, especially when it comes to data, you know, the data team. So that's super cool to hear. So taking a little bit now, you know, further, so help us understand how does the Zendesk stacks look like? You know, what kind of tools and platform do you guys use internally?

Yeah, I think it's honestly, when I joined, I was very surprised to see this because again, it's many data orgs and many people within the company thinking about data and thinking about how to actually build a platform and make it as accessible as possible. And so it goes in many different direction, but as you said, the main component is central foundation data platform, which is basically on AWS on S3. So we have a main, it's two components. One is the data lake and the other is data hub. The data lake is basically datasets on S3 that are managed by Apache Hudi and that are offered to consumption with Athena.

So you can query it via Athena and the tables are on the glue catalog. So the typical AWS stack for managing data lake. And that data comes in from basically change data capture from the product databases. So the bin logs and from the bin logs, you extract all the changes and events pushed to data lake via Kafka. So it's let's say it's yeah a typical CDC process in which the advantage of using Hudi is that we can actually manage the data at row level. So for GDPR, for example, and other reasons, Hudi offers you capabilities to actually ensure that you can delete, for example, one specific row without any large repercussions to that and you can actually modify the row level information in a very efficient manner. And also we use lake formation to actually manage, access to that data and row level access, and also column level access for security and also for data governance and on the second part of it. So data hub is so it's part of the initiative called platform data architecture. And the idea here is that product teams publish the data on their own. So it's product teams who publish events via Kafka then the events get consumed via Flink. And when they get pushed to S3, when again, it's Hudi and you have tables that are managed by Apache Hudi and this is the direction that the company is going through. So the idea is to get product teams to publish their own data. And so here it's easier to establish data contracts, to define standards, how to actually publish that data. What are the standards that every data assets that's published on PDA should meet? So it's a big portion of data mesh principles of how product teams should actually own their data and should publish the data that they're generating via their applications in a very standardized manner. And that's the path in which company is going. There is a very interesting blog that has been published very recently on the AWS tech blog by the head of foundation engineering that goes into the details of this platform. The initiative that’s actually implemented on AWS. It's very interesting. So, for a very deep dive into it, I totally recommend that blog. And it goes into the details of how this came together and how actually platform is doing now.

Sure. We will post the link of the blog along with the episode notes. So tha thank you so much for that. So Mahdi tell us a little bit about the, you know, uh, you talked bit of it in a sense that you have the change data capture process that kind of publishes those key product events back into your data lake and eventually Hudi and then Athena are you do you have your own custom built CDC infrastructure or using something like Debezium.

Yeah, for that, we're using the AWS product. It's called DMS. If I'm not mistaken. And that product basically, so Zendesk data is on around 2000 Aurora databases, Aurora MySQL databases. And to get the CDC from all those databases we're using the AWS product DMS, which basically works on capturing all those changes and pushing them into the dedicated S3 buckets on top of which they are managed by Hudi and on top of which you can access the data via Athena. So that's the part of a process on the foundation side, and that's how the product data gets into the data lake. And all of that is before, the team I'm part of EDA ,even starts operating on the data. So our team EDA we actually consume the data from the data lake. And then we get it into our own data warehouse, which is on Bigquery. And that's when we add other sources to it. So from what we get from AWS is mainly product data, but then you can imagine that you want to actually add other sources to that. So data you get from your CRM, the data that you get from your billing system, data that you get from Even your HR system. So you have a very number of systems that you actually want to add data from to that product data. And that happens on Bigquery on the data warehouse. And we move data from AWS into Bigquery again, using Kafka. So the company as a whole is very, it's very focused on event-driven architecture. So it's very rarely that you see data moving within Zendesk without having Kafka in-between or without relying on events.
Yeah.

So you, you talked about, you know consuming the product data using the CDC and you, you just talked about, you know, there are other data sources as well. So have you built your own custom connectors to be able to pull data from these sources? Or are you using some, you know, off the shelf tools like Airbyte or Fivetran?

Yeah, it's mostly custom connectors because again, I think with the company being very data-driven, we had to think about those needs and those initiatives before they were like, let's say democratized and all companies start to implement them. And so when you start thinking about it in 2017, for example, you have very very few options that you can pick from. And most of those options at the time, we didn't have all functionalities you were looking for. So if you wanted to do that six or seven years ago, you had to build a lot of stuff on your own. That was the case for Zendesk. A lot of it is internal components that run on Kubernetes that basically do a lot of a lot of the stuff that's currently today can be done with Airbyte or Fivetran. But for us, we had to start from scratch for most of it. So it's mostly internal components.

Amazing. And, and what kind of data volumes are we talking about? I would assume like given the huge scale of Zendesk, it would be huge. So any, any sense in terms what are the data volumes you're talking about?

Yeah, it's Petabytes, but it's like, as you said, in the intro, like Zendesk is a company we have round, I think more than 100,000 customers. And you can imagine the number of tickets that are being created on a daily basis. So we have billions of tickets. Usually users have a lot of complaints with the companies that they work with.

So there are a lot of support tickets that get created and yeah, it's billions and billions of tickets. And the total amount of data it's in the Petabytes. And again we're talking about data, so of product data that gets into the data lake. A portion of that is what comes into the data warehouse and where on data warehouse yet more sources, but just for product data created by Zendesk products is in the petabytes.

Got it. Got it. And one of the key things that we keep hearing, you know, in the data space and which is kind of one of the pitches for most of these ETL companies is that these connectors are hard to maintain, you know, they break and, you know there are change in API specifications, there are tons of things that keeps on happening. How often do you face this challenge? Like, you know, you are, you're kind of managing this internally. So how often do you face these challenges? See some connector broke and you have, you know, those pipelines broken how often that happens there in Zendesk. And what do you do to solve that?

Yeah, it does.

It does happen quite often because as you said the thing is when you build, like, when you work on data integration, you may start with maybe five data sources. But in one year, those 5 data source will become 20. So you can't actually build something without thinking about making it flexible, making it resilient because things will break. Things will change. And also you will add new types of data assets into it. So yeah, we do encounter those problems quite frequently. The idea is to ensure that you have logging that you have monitoring, but if something breaks, you are aware of it immediately and you can go out, immediately fix the issue and ensure that whether it's due to an API change, sometimes the API would be down for a period of time because that data source, let's say they're doing maintenance or they maybe updating something. So the API would be down. And it's, very hard to ensure communication on it and to ensure that you get the information when you need it, when there's that type of change. So let's say the brute force way of ensuring that at least when something goes down, you're aware of it just ensure that you're logging everything you have monitoring and alerting. So whenever something breaks, we have immediately an alert on slack that says Hey this task failed and you can immediately go and see why.

Tell us a little bit about how do. How have you built observability around these processes? So we are seeing you know, we are seeing two waves of tools coming in the data observability space, one is around data at rest, which is where once you have your data in the data warehouse you have set of tools that helps you, you know, maintain the sanctity of that data or check the sanctity of data, your tools like Monte Carlo Data, Anamlo there are tons of tools out there. And then there is also a set of tools that are focused more around observing data at motion. That means, you know, observing the data pipelines. Are you guys have any such tools around that or is that something that is, you know, kind of developed in house?

Yeah. So it's mostly for the data quality itself.

We, so we recently in the past year, worked a lot on that, especially on the data at rest did the monitoring and observability on top of that. Some of it is in-house processes because again very large parts of a stack are built inhouse. So we use airflow as our orchestration engine, but lot components.

And even how we use airflow is very customized to our own platform. So we have also customization on data quality. So as soon as we receive the data, and as soon as it's just into the raw layer, we perform data quality tests on it to ensure that the data is actually meeting the standards that we defined and only then you move it to the next step of your workflow. And then in terms of checking data, quality and observability, when the data is still in the streaming process or still in movement. That's something that we're still working on, at least with parts that I'm aware of and currently we're mostly monitoring breaking changes and if something breaks, we're aware of it immediately, then you can catch all potential issues. So it's a very big effort to be able to catch all potential issues as soon as they happen. So I think the bare minimum is that if something breaks, you're aware of it immediately.

Understood. And, Mahdi you talked about, you know, the ETL process at Zendesk and now keeping aside Zendesk, like from, you know, understanding from you as a senior data engineer, ,what are your thoughts around open source versus commercial solutions for ETL? Right? There's a, there's a new wave of, you know, open source ETL platforms coming up and the, basic pitch is, you know, we through democratizing the connectors, we build that long tail of connectors that, you know, is not kind of affordable to be, get maintained by a singular you know, commercial vendor. So what are your thoughts around that and how should any company, how should anyone earlier in their journey go about selecting commercial versus an on-prem on-prem or open source solution? Like probably Airbyte or Meltano

yeah. I mean, that's the, the eternal debate of build versus buy. I think it's a very delicate equation. You can't have an absolute right answer. It'll always depend on your use case. You just have to ensure that. Yeah, open source is free, but you also have an engineer that goes to actually implement what's open source. You have to not just implement it and deploy it. You have to maintain it. And you have to make sure that you actually can afford the resources and the time to actually maintain that product. And on the other hand, you have the financial requirement to actually buy something from a vendor, which will allow you to put your engineering resource into something else.

So it's a very delicate equation, which changes very easily, like the fine line in between from this to this has a lot of factors that are defining it. I think in general, for companies that are getting started or if you just want to get started with it. If you have very limited data engineering resources within the team might as well get those resources to actually producing value from the data, which is ensuring that you can actually build data products.

You can deliver data assets, whether it's data apps, or just curated data sets, or even allow your business users to access the data in a very efficient manner. And so that means that the more abstracted parts of the stack, where you're actually getting the data from different sources and you can start by just getting a vendor to deliver that capability for you and you get your team to work on the more business specific areas.
So for example, you have a lot of business logic in data transformation. You have a lot of business logic in how you deliver the data, and that's where you need people who are actually aware of all the constraints of your own team, but moving data from place A to place B usually it's not that specific to your case.

So if there's a vendor that offers all the connectors you need, might as well start with that, and eventually you can move to something that you built in house or an open source project, if you actually can afford those resources. But I think, yeah, I think just going with an open source project or a vendor based on the number of connectors, isn't always a good metric because you wouldn't need like 150 connectors getting started because as a small company, you would need maybe 10 basic connectors that most tools would offer to you. And if you have maybe one or two niche tools that you're using the cost of actually adding connectors for those tools, No matter which tool you're using, is very low compared to the effort that you would need to actually implement the platform as a whole. So I think that's the wrong metric to look at, especially if you're just getting started.

I think we need to just list what you can afford to spend in, in engineering time and say, can I actually dedicate this time to just building capability, but won't deliver immediate value to my customers? Or should I just work with vendor at least get started with that. And then eventually, maybe on the midterm, try to think about what would be the most efficient solution for this on the long term for me.

So I, I saw in one of articles that you mentioned that data technologies are going through a third wave. Tell us more. What do you mean by that?

Yeah, I mean, I think for most data engineers who are currently working on the current data stack. I think most of us started with the Hadoop era and Hadoop ecosystem and all of that, which when that started it was the second wave of let's say data technologies because people started working with data more than 20 years ago. So you have tools, but so the notion of the data warehouse on the notion of ETL and so on are, have been here for decades. And so initially it was a lot of heavy processing, a lot of slow processing to actually get the data into a consumable state. And it had very certain limitations, like the size of the data, the size of resources you can spend because your, the data warehouse is something that lives within your own infrastructure and you have to maintain it. You have to buy, if you want more data, you want to need, you need to buy new disks and so on. And then with Hadoop, it was the second wave. The notion of you can actually do this at scale. You no longer have limitation of one data warehouse or one instance for a data warehouse. So you can have an infinite number or very large number of just disposable hardware. On which you can actually run data processing, you can store your data and do it at scale. So that was a very important point for a lot of companies to actually say, Hey, all of this data, that I can actually leverage is something interesting. It's a big asset for me. So a lot of companies started betting on Hadoop on just trying to build that whole data platform internally Hadoop offered solutions to storing data at scale processing the data at scale with spark, but it was still very complex. It was still a long process to get up and running. And once you do you have new questions of how do you actually ensure data quality at scale? How do you actually ensure that the data that you're proposing at scale is accessible is democratized is easily queried by different types of users. Some of them want to use notebooks, some of them want to query data in an interactive manner. Some of them want dashboard on top of your data. Like how do you actually offer that? And then third wave with modern data stack is going even further. And instead of just focusing on those core capabilities that are now available, it's now tackling those small features. It's like, okay, you don't worry need to worry about data quality.

You have now a tool, that will do data quality for you. You don't need to worry about making the data that you have on your distributed platform accessible. You have a data catalog that will do that. You don't need to worry about, for example, ensuring that you capture metadata, you have a tool, that will get metadata for you and make it accessible.

So it's now each phase builds on top of a previous one. And with the modern data stack, it's basically just building on top of what we started with Hadoop era and just making the hard bits and yeah, the stuff that was complex to implement in Hadoop much easier because now it's managed services that you can basically get out of a box and you just need to plug your data into it.

And you know, one other thing that we have seen that has happened along with the evolution of the modern data stack is the shift from ETL to actually ELT, which is where you're pushing all the data into a data warehouse in a data lake without actually transforming it. And it is just sits there until someone needs that data.

And practically that data is of no use still then, and you know, we are all sold by these, you know, cloud data warehouses, like snowflake, where. You know, the storage is cheap, the computation is expensive. So you only pay for compute. So what impact do you think this have on the overall modeling and governance around data? Because now you don't think about, you know, You know, creating data and putting into a data lake, you just worry about, you know, making it consumable when you are going towards the, the compute part of it. What are your thoughts on that?

Yeah, I think, I think that's, it's a double edged sword because it makes things too easy and when that things get too easy, it's very easy also to just go into an abundance of creating tables using data in any way possible, because you can, and eventually you, if you'll leave your teams to do that, eventually you'll find what's now it's called the data swamp eventually. That's where you land because it's very easy to transform data to build tables on top of it. So I think we go back to point of standards and that you actually need to define standards, even if you can, theoretically like denormalize all your data and build any table possible. You still need to define standards, not only. So even if there are no longer compute constraints or storage constraints. Even for users, eventually you’ll realize that it's much more efficient and it's much easier for people to understand how the data is actually structured and what's in data warehouse, if you apply standards.

So I personally don't think that there’s one golden standard. You can go with the Kimball approach, you can even go with the approach of denormalized tables. You just need to standardize it. You just need to say, this is how we model data. This is how we're gonna. Go forward with defining our dataset. This is how we actually want the data to be visible to users. This is how our consumption layer on data warehouse gonna look like, and we need to respect those standards. And I think with that, it'll be much more efficient. First anyone in the company will be able to understand no matter what data model will looking at they'll understand. The current process, how the data is being built and okay, how can I consume this data?

And it'll be much easier for you to scale. Otherwise, if, if you just scale using the compute and storage capabilities because you can, eventually you you'll have the data, it'll be just very complex for you to consume it in efficient manner. And anyone who will look at the data warehouse, they will be lost.
They won't understand. Okay. Should they use this table or this table? And how is this being created or should they build something else? Sometimes they'll go and build something else and it'll be just more costs for you eventually, and also more complex to use data.

That's an amazing thought. That's an amazing thought, Mahdi.

So we are moving towards the end of our episode. And you know, we would like to wrap up the episode with a set of, you know, quick, rapid fire questions. You know, the questions that, you know, you don't have to think much about it. So let me start with the first question. Uh, tell me one tool that you feel you just can't live without. Tell me one tool in this whole data value chain, which you feel. Damn, you know, life would've been much different. Had it not been that tool?

I Mean, if you ask me the question, maybe five years ago, I would've said Spark because Spark was a complete game changer. Like I, as I said, I worked in moving data from data process from Hive to Spark and Spark makes life much easier, but right now I’d say dbt, like, especially for, yeah.

When you look at how SQL based pipelines were three years ago and there are things now with dbt it's, it's come like the quality of life upgrades you get are, are massive.

The next question is, we are seeing massive explosion of tools that are coming out of the modern data stack ecosystem, right? Where we have a lot of tools that are solving very small niche problems in that entire data value chain. What do you think is the future? What do you think the is the future having is having like separate tool for every single problem that you have in a value chain or you expect consolidation. Yeah.

I honestly think there will be a lot of consolidation because currently, like there's.

A lot of overlap on any aspect of the modern data stack. So you have your data quality tool that has dashboards to show you how the data is doing, but then you can get that same capability on a data catalog and even like how you discuss about data, you have every tool that will offer you those basic capabilities. And since it's basically a whole company that’s tackling a small feature, they're obliged to just expand. And with each one expanding out that core feature into other adjoint features, eventually there's just big overlap and very solid chances for consolidation, because it doesn't make sense for data teams to have like 20 contracts or work with 20 vendors. Eventually you'd want to minimize that. And I think that SaaS companies will figure that out eventually and we'll figure that consolidation is the only way forward.

Perfect. And then the next question is what's your go to place for learning new stuff about data, any particular blogs or, you know, newsletters that you follow and that you would like to share with the audience.

Sure. I mean, I think the first asset is the newsletter of my coworker at Zendesk and Ananth, Data Engineering Weekly, I think it's a very, I think it's great resource and I think the effort he puts every week in curating those articles are really just massive. Like I, I personally try to write maybe one article per month, and I sometimes struggle with that and you see that he reads all those articles every week and publishes stuff. So I think it's a very high-quality asset. And then honestly, I try to follow discussions on data Twitter. Usually that's where you get intrigued by different topics and bits of discussion. I think that Twitter offers a very digestible way of getting information. So we read a few tweets, you'd start thinking about something and then you go and do a deep dive into it, you read articles, you read blog posts and so on. So that's usually the workflow for me,

Yeah, honestly, there were quite a few to follow for example, Bar Moses CEO of Monte Carlo and there are also from Hightouch, Pedram. Who's also very active and who is the person who started most of the interesting debates. And when you start from their end, what I sometimes do is that when I find someone who's publishing interesting stuff on data stack, I just. Who are they following? And they start following those people.

Amazing. Then wrapping off with a last question. So tell us one thing that you love about your job and tell us one thing that you hate about your job

I mean, one thing that I love is seeing the impact of the things we built. And this is especially true, for example, within a company like Zendesk, you actually see that you are impacting decision making by the data assets that you're working on and the data pipelines that you're building, which is honestly it's I think there are very few jobs within tech, as you see that immediate impact at that scale. So I think it's quite rewarding and it's a very nice feeling. The thing that I hate, I think. It's something that the whole community is working on. That as data team you work with a lot of external stakeholders and you have a lot of potential things that can go wrong. Because the data platform depends on many things that are out of your control. Especially, if you, like, if you look at a more old school approach where you have a data platform team that's working on the data, you have data producers that don't care about what they're creating. It's very frustrating to have to fix something that you didn't actually create. So it's an issue that didn't start from your end, but yet you have to fix it and you have to just worry about it. And the team that actually created the data doesn't even care about it.

Thank you Mahdi so much for taking our time for this episode .I think so we had an amazing conversation and I hope the audience would enjoy our conversation as much as we had while even doing the conversation. So thank you so much for your time.

Yeah. Thank you. It was it was a lot of fun and it was great to, to discuss about all these data topics. Yeah. Thank you again for having me and looking forward to chatting again.

Managing ETL processes, data infrastructure, and data teams is already an arduous task, imagine doing all this in a very data-intensive organisation! In this episode, Greg takes us through how he and his team have kept things rolling at Canva. Greg walks us through what data platforms they use at Canva, ETL pipelines, data team structure, and much more.

Head of Data Platform at Canva

Data jobs at Canva

S01 E01: Understanding the data platform at Canva with Greg Roodt

Hello, everyone. You're listening to the Modern Data Show. For today's episode, we have Greg Roodt joining us from Sydney, Australia. Greg is the head of data platform at Canva, which is an online design and publishing tool with a mission to empower everyone in the world to design anything and publish anywhere.

Although Canva needs no introduction - it's used by over 60 million monthly active users and over 7 billion designs have been created on Canva. Before working at Canva, Greg was the CTO and co-founder of AirHelp, a claims management company that promotes and enforces passenger rights in the case of flight disruptions, globally.

Welcome, Greg. We are super happy to have you on the podcast and thank you for joining us today

Greg Roodt

So Greg, before we jump into, you know, the specific ETL part of the things, tell us a little bit about your journey from AirHelp to now at Canva

Yeah. So, my background is in I've worked in large data processing systems for most of my career and distributed systems as a software engineer I was fortunate enough to co-found AirHelp as a technical co-founder and CTO.

As you can imagine that business deals with a lot of flight data. So we were ingesting sort of flight data in real time into our platform. We were using Redshift at the time as well. So we were sort of entering that world of the warehouse. So that went really well. After a couple of years, I moved on went a little bit of traveling around the world and this exciting opportunity at Canva, popped into my inbox, and yeah, the rest is history. So about five and a half years ago, I joined Canva here in Sydney, Australia, and set up the, what was originally called data engineer and we built up the platforms from scratch essentially very large scale processing, as you can expect. And yeah have been fortunate enough to ride the wave of the modern data stack and hire a bunch of awesome people to help us build it.

Now that you talk about the modern data stack, you know, I think so the audience would love to know a little bit more about the data stack at the Canva. Like what platform are you using for, you know, various components of the data stack.

Sure. So, Canva originally data was a little bit siloed away from the rest of the sort of product. And my philosophy has been to have analytics as code and sort of shift analytics far left as possible. So we've been on a journey to do that. We used to have separate data lake separate data warehouse. And that used to be frustrating between the data scientists and the data analyst, trying to work on like they needed the same data set, but one was in S3, in a lake and one was in a warehouse platform.

And over the years we have consolidated everything into one unified modern cloud warehouse. We use snowflake. So snowflake is our data platform. We are heavy users of dbt. So dbt is the sort of central nervous system of all of the transformation that we do. To get data into the system, we use Fivetran for our third party loading and we have our own custom built first party loaders for our primary data stores and events. So we built a first party data loading, service and system that loads into fivetran sorry, that loads into snowflake fivetran is also going into snowflake. And then from there the warehouse platform team has built a framework around dbt that enables our analysts to model and manipulate and transform the data through various stages.

And then on the consumption side we have Looker for our sort of high quality curated business metrics. And for the more ad hoc analysis and exploration, to empower users who know SQL we've got Mode analytics connected as well. Finally we do a bit of reverse ETL. So we do operationalize some of this data and we use Census to sync that data out into various third party destinations as well. So we've kind of got that full end to end data stack.

Yeah. Thanks, Greg. So there, there's a school of thought that primarily believes that, you know, reverse ETL is kind of an extension of ETL. So would love to understand your journey in terms of, you know, how did you manage syncing the data warehouse back to those operational systems before using a reverse ETL tool?

What was the process around there before, using a particular tool?

Yeah, totally. it, used to just be a bunch of like bespoke scripts and it was done a lot less frequently. You were either you would have a silo, like maybe your events would go into segment or something and then segment would forward it on.

So it was very ad hoc. You were not really able to operationalize using your enriched data as much as you are now. So before that we really had very little operational usage of our data. It was very like, okay, let's build a dedicated pipeline. and we would always have marketing teams trying to say, well, can we get LTV into the platform?

And it's like, oh, okay. Someone's gotta write a processing job to specifically do that. Now that we're able to use the enriched data in the warehouse, you're able to build a table that's like customer and LTV. And then you can sync that wherever it may need to go. So, yeah, it's a very nice new it's so wonderful how easy it has become.

uh, Yeah.

a couple of seconds back, you, mentioned, you know, analytics as a code. And I think so that school, thought emerges from your background from software engineering itself. And it's, a lot of companies these days are, trying to apply the principles of software engineering when it comes to data engineering.

What are your thoughts around that? Is, that a too far fetched or too much of a stretch in when it comes to applying software engineering into the data engineering?

I definitely think it's the way that everything is moving. I think we've seen similar with the whole DevOps movement and, the sort of there used to be historical silos between operations and software engineering and throwing things over the wall.

And even with data engineer, And analytics before. I've, we've actually broken down the silo as much as possible. So we no longer at Canva have a data engineering specialty. We've really tried to embrace analytics engineering. So the end to end enablement We do have analytics engineers who look after the more platform centric piece, but we also have analysts, engineers embedded into the product groups.

You know, who, work closer with the subject matter experts to do the work. we try to keep everything as code because it, gives you the history. It gives you that ability to have an audit record. It gives you the ability to like, share your work. It seems to be the direction that everything is going.

we had infrastructure is code. I think analytics as code. We have, we're seeing data quality rules as code. We seeing data transformation as code. So I think. The days of, you know, visual UIs to wire up your systems and things like that, are, behind us. I think, people coming out of universities and things like that are, learning how to use Gits and, you know, everyone is becoming more and more technical.

so I think. I think software is eating the world that's saying. And, and I think it's similar for, data and analytics.

Amazing. then, you know, Greg, one, one specific thing that I would love to get, a little bit more of your thoughts around is, when it comes to specifically around ETL, you said, you guys use fivetran as a, kind of a major platform to be able to sync data from various operational sources and put it into a data warehouse.

one of, the recent trends that has been emerging in the industry is around, you know, open source platforms for this data integration. we have tools like Airbyte coming up, which is basically, an open source, ETL platform. That's kind of competing with Fivetran and one of the. you know, the key, value proposition when it comes to open source is, tackling the long tail of connectors when it comes to, you know, building those data pipelines.

how often do you find yourself in a situation where you would want to integrate some of your systems into the data warehouse, but you don't have a ready connector and what happens in those cases?

Sure. So, certainly I'm certainly of the opinion that there, that engineers shouldn't write ETL there's that there's that blog post that came out from, I think it was stitch.

and we resonate a lot with that. we, initially thought because Canva's very engineering heavy and product heavy organization. We originally thought. Let's write our own integrations, right. We are software engineers let's do this. and we did, but we ended up writing just three. We wrote three.

it was a enormous amount of effort for these engineers. It wasn't really very interesting work for them. and there was like, well, we've got three. It's gonna take us forever to do. A hundred. so quite quickly we got business support to justify the expense of purchasing a service provider, such as Fivetran and we quickly switched to Fivetran and we were like immediately able to connect like 20 connectors with like hardly any effort. and that mindset sort of enabled us. So presently we have like beyond 70 connectors through fivetran and then for the, for the long tail, we only have about five or six that we have had to write.

And it, it is incredibly easy, with, fivetran And I think also other providers, because they have like a Lambda connector, so that they'll call you all Lambda. So yes, you have to write a bit a code, but you don't have to do all the scheduling and the monitoring, etc They, provide a little bit of a framework.

So we do still have to do that. But I think the expense of a paying for SaaS product, like Fivetran or the other vendors that are out there is well worth it, if you compare it to the engineer's salary. So we are quite a small team powering quite a large organization. So we, think Fivetran is value money for our, for us.

we are able to tackle the long tail, but we might just be in a unique situation. Look, we're quite a. Like a SaaS company with all the typical SaaS connectors that you might expect, the salesforces the Google ads, et cetera. So we've got maybe. Very common connections, perhaps in other industries, you might have more like very unique needs.

And then perhaps you might need to build your own complement a paid product with, a bit more run at yourself, open source, sort of services. We, didn't have to do that. we, tackle the, custom stuff using the Fivetran stuff. And so far everything's been, worked really well for us.

that's a very interesting thought that I, didn't know that where what you can do is if you have a very specific connector, all you can do is define the business logic in a Lambda and let Fivetran take care of the whole scheduling and infrastructure. So that's a very, interesting thing.

Amazing. So, tell us a little bit more about the data volumes. Like, you know, I believe in terms of the scale at which Canva operates you know, the amount of data that you have been kind of managing or kind of moving around from places would be huge. you, mentioned you guys are on snowflake, right?

So right now you're on snowflake. And, you know, as with any other cloud data warehouses, it's you pay for the compute? the major cost, when it comes to cloud data, warehouse is come, comes from the compute, How do you manage these exploding costs specifically when it comes to the cloud data warehouse?

I'm not even talking about the Fivetran piece of, piece of, this whole picture, but specifically around cloud data warehouses, how do you manage those costs?

Yeah, look, it is, a constant challenge. snowflake, these cloud waves are these consumption based warehouses where they charge you for the compute.

you have to, you do have to keep your eye on it. there are controls you can put in place with snowflake that sort of alert you when you're using too many credits, cetera, you can put quotas, onto specific users or specific areas. so we are on a journey. we are trying to keep our costs control.

We're starting on a project to attribute costs. more fine grained. So two specific business areas at the moment, we've got very broad buckets of like, okay, this is for experiment analysis. This is for business intelligence, but we'd rather really like to go. This is specifically for the marketing department.

This is for the finance department. So we're gonna. Move to a bit more granular monitoring and you can do that with tags. So we're gonna tag, we're gonna tag our data models and we're gonna be able to, attribute the cost more effectively. But yes, you, have to make your clusters auto suspend.

you have to make your BI tools point at the sort of more aggregated layers. Don't let the BI tools point at the raw layers that you know, will be slow to query, et cetera. So yeah, you have to keep an eye on it. It is a challenge. however, you do have to. What you really need to do is if you can sell a positive ROI story on your data group, that helps.

If people can see that, yes, these tools cost a lot of money, but without them, you know, think of all the use cases, we're powering at Canva We power our machine learning platforms. We power our business intelligence platforms. We power our experiments analysis, our AB testing framework as processing data through snowflake.

We've got the reverse ETL stuff. So. You, you need to sell that story. You do have to keep an eye on it. it's not easy. However, like the engineering time that's that you save by being enabled with tools that are a little bit easier sort of end to end tools like dbt where your analysts can actually almost build a pipeline.

Right. that's kind of gold because pre snowflake we used to have EMR jobs and, you know, the, you know, running over S3 and that's like a whole lot of heavy lifting with AWS infrastructure and Terraform. And like when the spark job fails, it's like, whoa, that's complicated. So yes, the things get expensive.

we are processing, we've got petabytes of so we we've put petabytes of data in snowflake you do have to archive data, get it outta snowflake, you know, keep an eye on your, compute. but as long as you're getting business value out of it, I think, you know, if you can say you spend $1, but it returns you $1.50.

It's easy to get the business case.

Wow. Amazing. So, you know, I think that this is for the first time I'm hearing, attributing cost in the data teams, because that's a very common concept that you would hear in the marketing side of the things where you attribute the marketing cost. So that's, a very new concept that I've heard, like, you know, attributing cost for the data teams.

So, let's dive a little bit deeper into that. when you say you attribute cost, when it comes to like, you know, associating the data processing cost for various teams, do you just do it at the data warehouse level or do you attribute that cost across the value chain across the tools that is there, in, your data engineering processes?

At the moment, we're just looking at the, big buckets, which would be the, processing cost and the warehouse. And that will be the easiest place to do it. So, that's where we are starting. We see that in a similar way, as you might see, your AWS infrastructure, your cloud infrastructure, where like each service team or each area might be like, okay, you spend.

X thousand dollars on AWS, you know, so each of those, it's almost like this chargeback infras, you know, your infrastructure team can't justify spending that it's like, well, why are you running this computer? It's not actually for infrastructure teams. it's for the marketing purpose or for the product purpose.

So the data platforms are similar. Like we're running a data platform. We're not necessarily the subject matter experts of all of the data flowing through our systems. Right. We, we help connect things. We help things run on time. We help keep them operational. But if someone asks us about a specific column, We'll point them at the data catalog, for example.

So you, for us to justify like, oh, you're spending so many thousand dollars on snowflake, we need to be able to say, well, that's actually because of usage, that's driven by the marketing team or the sales team or the education team. So yeah, I think we're presently just doing it. at the snowflake level, the data warehouse level, but you're right.

Like in perhaps in future we could attribute the Fivetran costs that way based on the, who owns this connector, is that connector or so the credits that are coming through Fivetran that's actually a good idea. Maybe I should do that. we could, we could attribute that connector cuz those connectors also cost credits and money.

we would be able to say, well, that connector is actually coming from the marketing team. Do we think that the marketing team is using their. Wisely. Great. No problem. Or, oh, maybe we should reconsider. So yeah. presently, we're just looking at the big cost, which is the, this is the warehouse cost, but you could even do it for your.

Data loading and your data unloading through, you know, Fivetran or, Census as well. So,

so, Greg recently, there is a lot of talk around, you know, the data trust issue and the quality of the data and the most common complaint among the data consumer is that data do not trust the data that comes in and, Part of the, the whole process around creating a value chain around data is focusing on the quality itself.

Right? So I want you to help our, you know, listeners understand how do you handle data quality when it comes to both data at rest, which is once the data is there into your data warehouse, as well as. In terms of monitoring the quality of data when the data is at motion, when it comes to your, you know, ETL pipeline.

So help us understand the processes and your practices around data quality for both aspects.

if we're honest, we're still very early on this journey. we are focusing more and more on that. It's really challenging. at the moment we've got, alerting and monitoring. In our loading for our, for the data loading framework that we built ourselves, we've got datadog monitoring that all of those, extraction processes are running.

We've got, you know, observability over our Kubernetes clusters, et cetera. the Fivetran You get the error notifications when there's a failure to load. So we've got some simplistic stuff there. We really wanna look into the, reconciliation checks to make sure that the data that's in the source is actually matching in the destination.

So we early there, we've got some. Potential things that we're going to look into, but we're presently not very mature there as it's moving through the, transformation pipelines, in dbt, et cetera, we do have some, basic DBT tests. So we use dbt tests a lot, but there's still sort of, not really at a high semantic level.

So they're more at the technical, control level, like making sure things are unique, making sure. It's a key that actually matches at another table. So we've got some basic technical controls that we do run, into the pipeline, which do help. Like they actually are quite effective. They do catch things.

but we do want to put a layer on top of that, where we do a bit more semantic, meaningful tests, or that, when you look at things like, great expectations and things like that, maybe dbt great expectations to sort of plug in. So. we do wanna focus more on the data governance and data management aspects, as we mature, but presently we're, struggling just like everyone else.

the explosion of the modern data stack, the ability to do all of this stuff so quickly, Sometimes you're not exactly sure what's flowing through through the system. So, all of the start observability tooling, all of the start quality tooling is definitely an area that we're going to, invest in next.

Amazing. you know, next thing I want to talk about Greg is, you know, specifically when it comes to the data teams and the way data teams are organized within various companies, there are a lot of business stakeholders who are always like data engineers after data teams. Like, you know, I need to get this done.

I need to get this data. How do you. How do you manage those expectations specifically when it comes to, you know, justifying the ROI for, for that particular request data request that has comes in. have you made any attempts or any thoughts in terms of building self-serve analytics platform for various stakeholders or do a lot of business requests directly come through the data teams and which is where you would have your prioritization processes, help us understand through that process.

I think we struggle just like everyone else. Probably. we have a federated model. So we don't have a central intake. We do have a federated model. So we're, we hire data analysts into the, teams and groups where they can be most effective. So group leads are able to hire data analysts, into their groups.

and recently I think over the last year or so, we've also been pushing Federated analytics engineers. So sometimes you pair data analyst with an analytics engineer. If it's a particularly data heavy area, marketing's quite a heavy area, a lot of specialized transformation, attribution modeling, et cetera.

So we're doing similar there. So we actually allow, groups, departments effectively to own their own head count. And. Analytics engineering has been successful at canvas. So they're actually quite in demand. We actually have, we're continuing to bulk up the sort of analytics engineers across the organization.

they can tell the central platform teams what's working well. What's not working well. So managing the stakeholder expectations is still hard though. You know, stakeholders. I want a dashboard. They don't realize that like, well, behind the dashboard is, you know, creating the dashboard's one thing that's hard for the data analyst or the business intelligence analyst who whoever's creating the dashboard.

That's, skilled work. It's hard. And then it's like, well actually getting the data there in the first place. If it's not there, it's like, oh, that's, that's a whole bunch of transformation. Do we have the data at all? How do we model it? So I think. That's just a education thing. I think that's an empathy thing.

I think most people have come to realize that you don't ask an engineer to create a web app or a page like the next day. It's like, that takes time. They think, I think they understand that it takes time, takes work to do, to do that. I think that I think the understanding the data literacy needs to improve to a level where people understand that, oh, actually, It takes time to create these data products, these assets, you know, they need to value them.

They shouldn't just, if they just need the number for like a, very low priority reason, they should say. So, they should say, oh, I just need a guess. I don't really need a actual report. So, that's just challenging. I think you just need some advocacy in your company. We, certainly, I'm not saying we have it.

Perfect. We don't have it. Right. it's we, some, stakeholders can be challenging. we are under resourced just like everyone else that can never be enough data people so we're always hiring data analysts. We're always hiring analytics engineers. never seems to be enough so it's, just hard.

but it's, it is getting easier and easier. I mean, the cloud warehouse has made this explosion work, so dbt the cloud warehouse. So it's almost like a victim of its own success. The more data you have, the more answers people want. Like a, it just makes. yeah, you provide some data, then there's just more questions and then you provide more data.

So it sort of accelerates, here is no end yeah.

Yeah, no amazing. I mean, cause I actually have one question around in terms of what kind of skills do you look when you are, you know, when you're making a new data hire And before, before we go into that, I have another question is, help us understand how your data organization is structured.

So you mentioned you have a global data platform team, and then you have federated engineers, help us understand a little bit more about the org structure specifically around data.

Sure. So we, have a federated structure. so we have a central data group that, looks after the data platforms looks after sort of central key metrics and insights and looks after sort of data governance and data management, areas.

so we've kind of got a central, I guess, but more platform. Centric view of, how to enable the organization, holistically. So not only, Canva also some of the acquisitions that we've made. So, I sit in that org and I look after sort of the more technical teams there. we have, specialties, what we call analytics, engineer, machine learning engineer, and then.

Traditional software engineers, frontend, backend, et cetera. these roles federate across the organization. So you might be an analytics engineer in one of my platform teams, you know, building frameworks around dbt setting, best practices around dbt, making sure that the dbt jobs run on time. So they're sort of data agnostic, you know, you sort of solve the problem in an agnostic way so that it can be federated out to solve every problem. So they're more generalists and more technical minded folks. and then we, we federate those roles out into other teams in the organization. So, the content and discovery team, where there's a lot of content at Canva there's analytics, engineers there, helping them build their own insights over their own data.

There's folks in the marketing space, the sales space. So we're hiring, federated roles, across the organization. I look after three key areas. So like we've got our own event processing system. So click stream data sort of data, in motion. So that's sort of more traditional software engineering at scale, you know, thousands of events per second.

then we've got the sort of warehouse area with a data at rest. Data comes in, gets processed, gets transformed, and then we've got a machine learning platform as well, which is operating like training platforms and serving. For the organization. so for this, podcast, it's probably the analytics engineers and the warehouse org.

but we also federate the ML engineers as well. So we've got ML engineers and my team building the ML platform. And ML engineers in the product building ML powered product features. So, I think the federated model probably applies at most sort of medium to large organizations. You get to a point where you can't become a central bottleneck and you need to begin to build a, have a platform mindset.

There are, it's not perfect, but it seems to be, it works pretty well for us. It seems to be a model that kind of applies at at many places. It seems to be the way that I'm seeing most modern organizations, moving and it does enable the sort of level of self-serve because once you've got these building blocks together, you know, the fivetran the snowflake, the dbt The BI tool, you can basically take that stack and put it wherever in the org and sort of replicate it like, like software, like everything, software defined. It's like, cool, got a config fall that lets you set the stack up it's you are able to sort of copy paste it to some extent. the challenge is making sure that each, implementation is sort of following the same patterns.

So you do need a way to in ensure that everyone is continuing down the same paved. road which is what we call them a Canva. so we solve a problem once ideally, and then replicated across the organization.

Perfect. Amazing. And now, so that brings to the next question, like in terms of what skills do you look for, in specifically when you're looking to make a new data higher and specifically like young data hires, any specific set of skills that you look maybe like the technical skills or the soft skills that you look for, any new data.

hire

Yeah. So Canva has quite a structured engineering hiring pipeline. So we do build pipelines for the bespoke for the role. So for analytics engineers, we have a specific sort of set of interview questions and challenges that they go through. you know, we have some technical pre-screening questions and hour long sort of coding interview and then a few sessions of you. know Warehousing data platforms, data modeling, et cetera. But we also do have the, soft skills sort of interview, you know, the communication skills, the leadership, the strategy thinking. so we do have that as well. So we want all rounders. we're generally not looking for any particular technology, so we don't need people to have worked with AWS or snowflake or even dbt. We talk about fundamentals, for the analytics engineering side, it's, Python coding. So like, can they write basic scripts and, sort of, you know, general Python skills are, valued here. and then classic warehousing, you know, like. How do you move data through a system, ELT, you know, how, what are the best practices for modeling data?

Can you do dimensional modeling, you know, can you solve this business scenario? So we put a business scenario, in front of the candidate and we work with them during the interview session to solve the, you know, solve the problem with SQL or, however they'd like to solve it. So, yeah, technical skills, like I believe that analytics is code.

So for us, everyone sort of needs. To code. self-taught is fine. doesn't matter where they studied or what they did. but we do, value the coding skills. So we do put technical, challenges in front of.

Amazing. So, and, are you guys currently hiring? We're

Always hiring . Yes.

we are fortunate enough to be hiring we've not hiring as much as we had planned, but fortunately we, haven't had to stop or anything like that. So we are just, we've reduced the rate at which we're hiring while we keep an eye on the market. But, yes, we are still hiring for analytics engineers.

So if, anyone's listening and they, wanna join, please reach out.

we'll have the, link to apply for the open roles in the show for net as well.

Yes. So I, I think so, you know, that's pretty much it from our side. you know, Greg, thank you so much for answering all of those questions in such an amazing way.

And I hope, a lot of our listeners would be able to learn in terms of the data practices and the processes that you have around Canva and make world a little better place. So thank you very much for joining this show, Greg. And it was such a pleasure talking to you.

The
Modern Data
Show.

Episodes

TheModern DataShow.

Episodes

The
Modern Data
Show.