Lauren Balik, who runs Upright Analytics and is a leading data consultant and investor, discusses why she believes the modern data stack is flawed and the three factors that affect the cost of a data platform. Balik also compares building versus buying a data platform and recommends an OLAP database in the cloud for small companies. However, she thinks centralizing data out of a line of business is a mistake for larger companies. Balik does not anticipate consolidation in the modern data stack and thinks that large language models such as GPT-3 will be crucial.

Owner

Host

S02 E05: What's Fundamentally Wrong with Modern Data Stack with Lauren Balik, Owner at Upright Analytics

Welcome to the Modern Data Show, where we explore the latest advancement and insights in the world of data. Today we have a very special guest, Lauren Balik, who runs Upright Analytics and is a leading data consultant and investor. With her company, Lauren and her team are solving the toughest data challenges and empowering data-driven cultures, helping people fall in love with their data.
Join us as we dive deep into the mind of a true data expert and learn about cutting-edge solutions and strategies she's bringing into the industry. So sit back and relax and get ready to learn all about the world of Modern Data with Lauren Balik on the Modern Data Show. Welcome to the show, Lauren.

Aayush Jain

Lauren Balik

Okay, so Lauren, let's start with a very quick primer on Lauren. Would love to know a little bit more about your journey and how you got into data and your story of starting Upright Analytics.

Yeah Absolutely. In college, I studied economics. I fell in love with data there and applying it, I was using everything from MATLAB, learned some SQL, and did all that in school.
Yeah, I went to the University of Virginia. So it's been about yeah like I was doing that a little over 10 years ago now. So, time flies, but love data, fell in love with it, graduated, and then got into sales, which is my first job outta school, I was selling software and working at a tech startup selling mobile messaging software.
Nothing to do with data, not applying the things I like to do at all. And it was a very very interesting learning experience. I learned about the tech startup space and everything, but after about a year, I had to get back into it. Rather than prospecting people and doing the phone calls and doing, I guess what today we call them SDRs doing that type of job right out of school got back in the data.
I've worked with organizations over the last decade or so from, some of the biggest names in the world, the Fortune five hundred, large electronics companies down to tech startups. Have seen, the rise of Tableau, the fall of Tableau, the rise of the cloud data warehouse, the what might be the start of the fall of the cloud data warehouse now.
So, have seen it all. I like to be pretty pragmatic. Keep my head screwed on right about things and separate buzz terms from actually providing customer value to very different things and yeah, at Operate Analytics run a very small lean team. I would say over the last year and a half, we have mostly just been doing open heart surgery, is how I describe it.
And so if you've ever seen a movie, like Pulp Fiction they have to call in the wolf at one point to clean up a big mess. So, they call up the wolf, the wolf comes in, you go over here, you don't say anything, first, we clean this, then we clean this, then fix this up.
And I've been doing that for I've been running my company for a few years now, but last year and a half has been a lot of that. And I think it's a function of a lot of the over-eagerness, overspending, and maybe not hiring people in the right roles. It's a function of a lot of things, but that's what we do.
And, work with organizations large and small.

Amazing. And let's address the elephant in the room. You're not a big fan of Modern Data Stack. You already mentioned Tableau is dead. You already mentioned cloud warehouses are about to be dead. Let, before we go into that first help let's bring, let's be on the same page.
What's your definition of the modern data stack? Which modern data stack we are talking about?

Yeah when I say the modern data stack, and I think what most people think of is the idea of, a centralized cloud data warehouse that operates as the computing centre and a single source of truth around which sit, you know, outcomes that range from potentially machine learning.
Usually, it's analytics and BI and there's a heavy lean towards, I think, SaaS applications that plug in versus coding especially, when you see ingestion jobs, ETL. So there's a definite lean towards that. And so that's what I would define as the modern data stack.
I can put the lakehouse in there, some of the things that are going on with a lake house type architecture. But, I think it's all the same at the end of the day.

This story sounds good. You have the central data warehouse where you have the central source of business knowledge, and then you build applications like the data applications on top of that to be able to support the business decisions.
What's wrong with this? What's fundamentally wrong with the modern data stack, or where do you think Modern Data Stack fails to deliver?

Think if we look, just broad lens, if we take off our engineering or analyst hats and just look at the broader state of how did we get here, what's happening, et cetera.
The whole thing is based on the NRR or NDR, net dollar retention of Snowflake, Databricks, Google Cloud BigQuery, and the Google Cloud Ecosystem it's all based on the net dollar retention of these compute centres. So you store your data, then you do stuff with it, you make DAGs, you make SQL statements, et cetera, in a variety of different ways.
And because like if you look at, and this is broader than just the modern data stack, you can look at monitoring and observability stuff that is in the broader engineering world and broader infrastructure world like Datadog, HashiCorp, they all have great NDRs. They all have great, every year, from the previous period to this period, customers are spending, 130% more, 140%.
I think snowflake's gotten up to 178%. So these customers are spending more and more every year on a lot of these cloud products. And when we're talking about the modern data stack we're talking about the compute centres here and no one has ever brought this up I think, but if you talk about if you have a credit card and you're paying 30% interest on it, that's a bad credit card.
Like you're getting screwed over a little bit by the credit card company. It's high interest. And if you compare the net dollar retention of what some of these companies are, and let's be honest, it all centres around the computing center of the cloud warehouse and the modern data stack world.
That's what drives everything else forward. And so I think right off the bat like there's a little bit of, I don't know, maybe a misplaced incentive around what customer success looks like versus how much a customer is paying. And there are a lot of incentives that may not line up with perfect customer value in that second year, in that third year versus what they're paying.
Okay.

And what do you think is the biggest culprit of this? Are these, this is the data warehouse itself or the ingestion pipelines that are dumping an insane amount of data into the warehouse and this whole shift from ETL to ELT? What's the main culprit there?

Yeah, the rebranding ETL is ELT is funny because it just moves instead of processing the data. I don't know, in Spark or wherever you're going to process it beforehand or define your schema, you're just dumping it somewhere else and then defining it later. So they own the computer there. But you know, overall, I think the main culprit in this is that a lot of companies aren't educated on what it looks like to succeed.
Snowflake for all the great things that they've done and in Google Cloud as well as BigQuery, they've lowered the bar and lowered the entry point of what it takes for a company to get started. They want the data now. They want to build some analytics.
They wanna build some reports and things, and that is that's great, but if you don't design stuff correctly upfront, you're taking a payday loan at 30% interest, or in the case of Snowflake, I don't know what are they, 60% NDR or something. You're taking a 60% loan on average and 60% like interest.
And this adds up like it adds up in the second year. It adds up in the third year. A lot of it comes down to SQL. What we've seen here is the rise of SQL, in the last couple of years. SQL is easier to learn than an object-oriented programming language. Using declarative SQL to define your data schema in Snowflake in a database, yeah, it works fine.
It works great. If you're not writing optimal SQL and you're doing things like joining on high cardinality data that can add up very quickly if you're doing like looping functions or nested loops in SQL. That's gonna add up and a lot of people don't think about this, but I think a lot more like both individual contributors, data directors, and managers need to think about this more about the exponential growth.
If you're running some kind of function, you know right now, like it can cost you like, I don't know, one snowflake credit next period, it can cost you two snowflake credits when your volume goes up and you're running it. Now, if you're doing it linearly, it should take you three. It costs you three credits next time.
But if it's growing exponentially, it's not gonna cost you three. It's gonna cost you four. And then next time it's gonna cost you eight. You're gonna double it every time cuz you're making all of these, N-squared types of solutions, and I've seen this pretty constantly. And the same with indexing data or figuring out how data's partitioned.
A lot of basic stuff falls by the wayside because of the desire to move fast and moving fast is good. And moving fast is the name of the game for a lot of these tech companies in the last few years when it's all been run, run, run, but now, it's time to optimize a little bit.
And, I even wrote a piece recently on my Medium about, Snowflake as a table game. But the Google Cloud ecosystem is a casino. And, if we're talking about things that are wrong or maybe look off one of the funniest things here, and I love Google. I like BigQuery a lot, and I like their ecosystem.
I like all their solutions and use them all the time. But one thing I've gone over throughout my career is with these tech companies like they'll start using Google Analytics, they'll start doing Google paid ads, and like that. If they're in consumers, like they're selling consumer goods to individuals, they're definitely using Google Ads, Facebook ads, and everything else.
Google ones can be big. Same with some B2B but they're using all these Google, they're paying. Google has lots of money for all these like applications, Firebase for their app, and they're collecting all kinds of data. And they get to a certain point when they say, wow, I'm spending so much money on Firebase and Google ads.
Like, why are we spending so much on Google Ads? We shouldn't be paying Google so much money. We need to get our tax to improve our customer acquisition costs, we shouldn't be, we should be targeting better customers. And so then what do they do? Everything plugs in nicely to the Google cloud platform like all of Google's products.
So then they hire machine learning analytics, and data engineering to work on Google Cloud. They get their tax down. They might pay Google ads less money, but now they're paying Google like BigQuery, and Google Cloud more. So Google, even though Google goes down over here on the ads. They go up on what they're taking out of these companies on the compute side of putting together dashboards, machine learning, and everything else to improve what they're spending on other Google products.
And I think that's just really funny overall and I think a lot of people have not thought about that fully. But it's definitely like what the game is. So, very funny stuff.

Yeah, so Isn't that in argument, in favour of the modern data stack where you're essential, unbundling a lot of these functions into kind of composable elements where you are giving the people a good amount of control in terms of how they want their data stack to be.

You're giving them more control and at scale, it works. If you're a big enough company, you can get good cost savings out of this. Most companies are not that big. If you're like, selling e-commerce, like if you only have a million customers or 2 million or 5 million customers, like actual humans out there who've ever like actually completed an order with whoever completed an order with your business and become, what's defined as a customer.
If you're operating at a level that's small. You're probably overspending on data if you're trying to tinker with these margins and fix your Google ads and fix everything else there. If you have a hundred million customers, if you're McDonald's or I don't know, Ford Motor Company or somebody then yes, it would make sense to potentially take this on and do all this analytics to tinker with I don't know if those companies are on Google, but you know, to tinker with, what you're spending on Google. Using Google Compute to do it

Right. And this also raises a kind of parallel question in terms of what are your thoughts on build versus buy. So basically what you're saying is companies rush in, in terms of buying managed services to get that initial friction out and get started with things.
But that kind of turns out to be a very costly mistake in the long run. But at the same time, the kind of engineering efforts and the development that you need to require that is non-core to what you're doing as a business. Isn't that counterproductive? Like you, you take the example of an e-commerce company, why would an e-commerce company, with less than a million users or whatever skill that you would define would invest resources in terms of building an internal data platform versus actually focusing on resources building that business itself? Isn't that a counterargument?

What you have to look at with these types of things is there are three factors at play here in what the cost is.
There are your cloud costs, your storage, and your compute of data, there are all the products you may want to add to it or not. Maybe if you wanna open-source stuff, you're not gonna pay. If you wanna pay for a bunch of SaaS applications, you can pay there. And then there's the HR costs the human resources like headcount costs.
And when you have one go up, you should have the other go down. So if you're paying if you hire more people, your cloud costs should go down because people can spend time optimizing. And if you're hiring more people, like maybe, you shouldn't be using so many SaaS applications because if you're hiring good people, they can build something and maybe it's more efficient.
But what we've seen here in the last couple of years is companies are spending lots of money on extra heads. The data team was two people, now it's six people, and now it's eight people. And they're using all these applications to move data around in various ways, ETL, Reverse ETL, and manage all the tables.
We could do a whole bit on the proliferation of tables. And so they're spending money in these three different buckets. And I don't think people talk about build versus buy the benefit of building is, you have ownership over your data more and you have more control.
But you're also probably gonna have to hire more people to do it or hire more expensive people anyways to do it. And right now we see expensive people, we see a lot of SaaS apps, and we see cloud costs going up. So all three are going up. And that's, that's been the challenge.

And, if you go to moderndatastack.xyz we have this page where we list the various categories that are there within the modern data stack.
Just our interpretation of those categories. There are roughly around 28 or 30 different categories out there and that seemed annoying that seemed to annoy you. And two questions there. One is, What is that one good thing that you really believe came out of this whole, modern data stack revolution that happened in the past couple of years?
What is that one true thing that, it's not a vendor-driven narrative, it's not a visa-driven narrative? That's one thing that the one needed and bam, someone solved it.

Yeah. I think a lot of people have learned SQL pretty well, and I think a lot of people who may have, their title may have been like marketing ops or sales ops, but now they're, I don't know, a data engineer or data scientist or analytics engineer or something.
A lot of people have been more educated on data, that's for sure. And, I think that's a good thing. So yeah, that's what I would say the number one best thing that's come outta this is a lot of people have gotten more educated. And the funny part about the categories to is.
You think about what's core, what you start with, and what you need to do data at your company. You need to collect data somehow. In the most simple way. It could be an Excel sheet of all your orders, and then you wanna make a chart and look at orders over the day or time or something.
But the more categories you build, like it all like pyramids up, like yes, in the modern data stack everyone, let's say has Snowflake or a BigQuery or Databricks or whatever. They've got a BI tool and they've got some way to get data in. So they've got 1, 2, 3, they've got the whole pipe there.
Then you move one step like those customers, some of them also have now observability, now or reliability, or whatever you want to call it. Now, some of them have a catalog on top of that. So but that's even a smaller number. And so, you get up to the point where, you know it turns into this dependency ladder because a lot of these, like observability for example, if they're just for the cloud, like they're not gonna have more customers.
Then there are customers who like using a cloud data warehouse, right? Like that number's always gonna be high, equal to or higher, like by default. And so the more assumptions you have to make about these categories and what's already in place and how big the team needs to be and how complex it is. The more gets riskier as an investment and, maybe not something that's considered necessary.
And maybe the TAM, the total addressable market, or the total serviceable market is a lot smaller than people think.

Fair enough. And you consult a lot of companies ranging from startups to Fortune 500 companies in terms of building a data strategy for them, right? If that would be the right term for that, what's your typical advice for them?
For example let's take a scenario, have this, the early-stage company that has just started to hit the scale. They come to you and say fine, we have found that initial product-market fit as a company, whatever we are doing as a business has started to catch off and now we need to start leveraging data to be able to grow even further.
What's your advice? What's the typical journey that you would take on with a customer in terms of helping them understand what they need? So what would be a typical flow for you?

Yeah. So, initially, it's what matters to the business in the next quarter and six months.
That's what matters. And in the last couple of years, with the way the economy's been and the way VC incentives work and the way these startup incentives are, it's all been growing. We don't have to worry about merging too much. Let's just grow. And so, you see all these growth roles like a lot of companies have like the director of growth.
Growth ops and these people are doing a lot of like ads and other growth hacking types of things to put in place. So if that's what's important is getting a growth metric to get to that next round of financing or to, get to that next stage you want to get to, then it's gonna be a growth initiative.
Now with a lot of larger businesses that, you know I've been around a while, and they're not, they've either gone past the VC game of growth or, they're just, financed by the cash they collect and by debt sometimes plus any equity finance that they have.
But aren't VC companies like these companies they care about their margin and they care about operational excellence and for them, it would be improving, CAC would be a very important thing. That comes up pretty constantly as it relates to growth. Another would be like these types of businesses when people talk about centralization, modern data stack, et cetera.
A lot of these businesses that have been around for a while and have over a 1000, 2000 people independent of industry, like they're gonna have some databases over here, they're gonna have some databases over there. They might be running a good they may have a good analyst who only works with Salesforce data and marketing data.
And none of that's connected to the product data, like a lot of these larger businesses. And I think when people talk about, how are things like in a different company versus the company where I work? Like a lot of these larger companies, they have something that resembles a data mesh in place a little bit.
And they often have decentralized analysts and they typically are able to serve themselves decently. Something like spin up, let's stuff, everything in Snowflake, or let's stuff it all into Redshift or whatever. That is something that they're not gonna want to do just because they're already humming along and they're already very much delivering on things and good enough works for these types of companies.

Right and if not modern data stack, what's the way forward? what do you think is the way forward?

I think, for a lot of these because the bar has been lowered to run an OLAP database in the cloud that's gonna stay. And that's a good, way for these small companies that are growing, maybe VC funded to get started and get some data in place and start delivering on some dashboards.
It shouldn't be the earliest thing you do but it should be, maybe when you get to 50 or 100 people somewhere around there, like that's when you should start pulling this together. So you know for those types of companies, I think this is gonna stay an operating paradigm that exists for the next couple of years.
And for larger companies, I think it's gonna stay decentralized like it always has been. And trying like trying to force the centralization of data and centralizing it out of a line of business is, I think a mistake in most cases. And it's just there are gonna be a lot of failed projects that happen.
As this, attempts to move up the market more.

Would you see a consolidation happening in the modern data stack? Last year was debundling. Do you see a bundling in the modern data stack?

So I think there's already a lot of bundled solutions out there. There's Keboola for example.
You can do orchestration in it. You can do your ingestion. You can manage the flow of data in and out of Snowflake or whatever you want. Same with a bunch of other solutions, Rivery is the one, Nexel is a great one. They have a lot of good enterprise customers. And so when people talk about these companies like, reverse ETLs and ETL companies and everything else, merging or acquiring each other. I don't think that's gonna happen because they're all way too expensive. They're all valued at $500 million or more. And outside of Fivetran and some of the BI tools, Most of these companies are under 20 million dollars in revenue.
So you're not gonna buy them for their revenue or their customer base really. And if you're not gonna buy 'em for the customer base, okay, we may have a complimentary feature, but you would just build that yourself. So I don't actually see this scenario of these companies, buying one another and consolidating that way.
I think that Keboolas,the Ascends, Nexlas the Riverys that already do multiple things, they're just gonna rise more and more. And I don't know what's gonna happen to a lot of these modern data stack companies. Like they all raise a lot of money in 2021 in 2020, and a lot of 'em have dozens to hundreds of employees.
Fivetran makes a lot of money. Some of the BI tools make, in the tens of millions or hundreds of millions. But everybody else, like they're gonna run outta cash by the end of the year. And so they're either gonna have to do a down round and, basically put all their employees underwater or they're just, gonna have to like, start letting people go.
And I think we're probably a couple of months away from that if you just look at the employees like employees are a big expense. And a lot of these companies, like if you're paying on average an employee, 200,000 US dollars a year with their salary insurance, the insurance you pay on them and multiply that out by you have a hundred employees.
You have 200 employees, you can pretty much see how the cash is burning down in that organization. And if they're not making their revenue to cover that and they're not raising new money, then I don't know what's gonna happen and I think they're gonna have to make some tough decisions here.

Let's just hope that doesn't happen.
Hope there is a happy way out here. But as we inch closer to the end of the episode, there's another interesting that I just would love, want to talk to you about it, which I saw in one of your articles about the impact of GPT3 on tech workers. Big question. What impact do you think GPT3 or these large language models would have on data, on the data industry what are the things that you see, are the most obvious ones to you?

Yeah so the most obvious here is, writing SQL that queries into an established model. And what I mean by the model is that the word model has gotten misused in the last couple of years by data science, analytics, engineering, et cetera.
To query into this data mart, data model, whatever you want to call it, that makes a lot of sense. And it makes sense because if you've defined the entities, this is a customer, this is a transaction, this is this type of transaction. If it's all well defined there's no reason that like GPT-4 when that's released later this year GPT-5 after that, there's no reason that like this code is not going to be able to be generated very successfully with maybe only a mild amount of handholding.
But the entity resolution piece is where the money is because you can go on Chat GPT-3 right now and say what is the female population in New York State, and because New York State, all the states are well-defined public data, the population that's female, they can come up with an answer very quickly because those entities are well defined and they're public.
Now, if you go into a business and say, how many orders did we do? And what was our net operating profit last quarter? Now you have all these metrics and these entities that have to be defined, and I still think that's gonna have a human element in a building for the next couple of years.

Yeah. Because you have orders underscore one table order underscore two table orders underscore final table. Yeah, okay that's very interesting. So, I think so we are seeing quite early results from, this whole natural language processing to sequel generation. And I think so the bigger problem is not just the ability to generate SQL from tricks, I think so that's a solved problem and that kind of works really well, I think. So the bigger question, as you rightly said, is how do you organize this knowledge that is there within an organization to be able to actually understand where to actually query this data? So that's a fair point.
Anything other than this then you see specifically for the data teams on the applications of the large language models?

Yeah, I think GitHub co-pilot is gonna continue to spread and popularity. It was controversial. I know Amazon I don't think it was AWS. I think it was Amazon's retail side of the house.
I think they actually banned co-pilot cause they were concerned about what code was potentially being collected by GitHub, which is owned by Microsoft. So, there would be a little bit of stuff to watch there, but I think that if the automation really saves so many human working hours, then it's only gonna continue.
And I know plenty of people who are saving half their time or more already using this.

Amazing. As a closing thought, what would be your prediction for the, and more than a prediction? What would be your top three insights for the data teams for this next year, what are those top three things that you would advise, data teams, and companies who have built out their data functions over the next year?
One obvious one is, as you've rightly said, cost. People would start to think about how they're spending on the warehouse or ETL credits and so on and so forth. What would be those things for you?

Yeah. And that's one I think and the human costs are what's most expensive, right?
And so that's why we're seeing so many layoffs. So I would, advise people and advise anyone listening. If you're, a data scientist, data engineer, or analytics engineer, like really figure out where do you fit into the company? As I've talked about a lot of these data jobs have become human middleware.
You're not on the application side, you're just in the middle. You're not doing analytics, you're just in the middle. The more in the middle you are, the worse off your situation might be. So I would encourage people to think about that. I think we're gonna see the rise of like Ops and so the last, decade we've seen developers DevTools, make a tool that does this thing and you can turn this snob over here and there's open source and we have a community around it.
And we've seen this everywhere, not just in the data world, but much more broadly. And I think that a lot, as a lot of these companies rethink this, a lot of these developers are just doing Ops. They're just doing operational things. And a lot of data especially is really just operations and the more steps and the more tools and the more people you have to get a number on a dashboard. If someone can come in and do that better and do that with Retool or Airtable or, instead of a database and a BI tool, they're gonna win cuz they're gonna be cheaper and they're gonna get it done more efficiently. And I think we already see this.
And I think the last one is I would encourage every company to do an audit of how many tables they have in their cloud data warehouse. This is where the costs add up because in every table you add if you have a medallion architecture where you have a staging, then you have another layer, then you have the final layer, the production layer gold, silver, bronze, or it's called a number of different things.
How many tables do you actually have ingested? And how normalized is it? Because one of the big trends was, your OLTP would be normalized and then you would ETL it and then your OLAP would be de-normalized to serve reporting and BI and analytics and everything. And we've moved away from that.
Now we're just copying things from an OLTP to an OLAP from different endpoints that come off APIs. Just bring it in de-normalized and then roll it back up and if you look at costs and look at efficiency and where your team's time is being spent. The more you make more tables and then roll them back up, the lot of that's wasted time.
And so I think a lot of people have realized that and have started to move to more, de-normalized structures as a goal and as a first goal. When they're building out their, Snowflake, BigQuery, what? Whatever else.

Interesting, thank you so much for that lovely piece of advice, and thank you so much for this so amazing conversation.
Lauren, it was such a pleasure to have you here on the show and I hope you know, we all learned quite a few things from this episode.

Yeah, thank you so much for having me. I appreciate it.

Ian Macomber, Head of Analytics Engineering & Data Science at Ramp, discusses the company's approach to automating finance tools and building the next generation of finance through data-driven decision-making. Macomber emphasizes the importance of cross-functional collaboration and embedding the data team into every part of the product engineering process. He also highlights the need for data compliance and privacy to be invested in every day and not treated as a one-time effort. Macomber warns against "Layerinitis," where teams prioritize quick solutions over long-term effects, and advises celebrating the hardening of code and inviting people into codebases to teach them best practices.

Head of Analytics Engineering & Data Science

S02 E04: Legacy to Modern: Transforming Analytics Infrastructure with Ian Macomber, Head of Analytics Engineering & Data Science at Ramp

Welcome to the Modern Data Show, where we explore the latest trends and technologies in data and analytics with some of the brightest minds in the industry today, we are excited to be joined by Ian, the head of analytics and engineering and Data Science at Ramp, a company that's revolutionizing the way companies spend less.
Ian brings a wealth of experience to the table, having previously led data teams at Wayfair and Drizly. Ian is known for his expertise in developing and improving analytics platforms and his ability to guide companies through fast pace changes for compliance and security. He's also an MBA graduate from Harvard Business School.
We are looking forward to diving into Ian Insights on data-driven decision-making, Modern BI stacks, and the future of corporate spending. So let's get started. Ian, thank you for being with us.

Thank you so much for having me. Looking forward to the chat.

Ian Macomber

Perfect. So Ian let's start with the first question.
Can you tell us a little bit more about yourself, your role at Ramp, and precisely what you and your team are responsible for?

Yeah. Maybe I can start a little bit talking about Ramp. It's a company that if you haven't used it it's hard to understand. So very generally Ramp is automating the finance stack for businesses right now.
So really building out the next generation of finance, automating tools. So that's corporate cards, which is what we are first known for. Now it's expense management, bill pay, accounting, integrations and very much a company that is designed around saving businesses time and money. That is truly what our vision is.
That's how we make decisions. That's how we think about products. And largely this has been a company that's grown super quickly. And what we're working on with the data team is figuring out how as we go from, one to ten to a hundred to tens of thousands of businesses, how we can start to take that data exhaust and help Ramp internally make better decisions, but externally also help Ramp, build products that we now have the right to, with all the data that we have to help save our customers more time and money.

Amazing. And how did that transition from the Harvard MBA to data and analytics happen?

Yeah. So the MBA was a bit of a detour. I started my analytics career at Wayfair. They had, at the time, legions of business intelligence analysts they hired outta school, and I was one of them.
And that's really where I learned about how a massive, soon-to-be public, international company, thought about data. Really sort of one generation of tech stack ago. So in my, first couple of years, I had experience with on-prem hardware things like Hadoop for clickstream data and distributed computing.
Having issues where, once you got to data of a certain size, you either had to not do the analysis or leverage customer infrastructure. At Wayfair got a little bit deeper into pricing and recommendation, machine learning. And when I left it was mostly because I felt I didn't want to get pigeonholed just into sort of the pricing machine learning space.
Especially for recommendations you can spend an entire career on that. Went to business school, I think with the intent that I would return to analytics. I think probably. Went in thinking it would be at a mega tech company, like Facebook or Google, and came out wanting to lead a data team and I had that opportunity at Drizly.
Drizly was my first experience at a cloud-native company. So it was founded in 2012 or 2013, so it never owned a server entirely on AWS. And I really remember thinking about, okay, this is not a culture of we build everything in-house. It's a culture of having to be very precise.
So if you think about the Modern data stack, right? I think what's valuable about it is if you take someone who's pretty talented and can read some docs and hack at some stuff and you give them a credit card, they can build most of an analytic stack. Maybe in the afternoon. And so that is really where I developed the modern data stack.
I think grew my career along with the trajectory of the modern data stack in these tools. Drizly was successful through the team substantially. There's a pretty big covid boom as it was Alcohol on demand and then came to Ramp recently and say Ramp largely was a company that's also been pretty successful.
and the same thing, really wanted to inject data. I think the first year was building a lot of the data, building blocks things that are just foundational to the business, right? Like how many website visitors did we have? Building out some of our first multi-touch attributions, and being able to answer product analytics questions.
And now what this next year is gonna be defined by is building things that are a lot more proprietary. So building out things that there might not be a medium article on or, a dbt slack talk on thinking about things that Ramp can build that are proprietary for the first time.

Amazing and can you help us understand how does Ramp's Corporate card differ from traditional card offerings that are available in the market?

Absolutely. And so there are a ton of answers to this question, but I'll just start with a simple example. If you think about maybe five years ago, you might have had an American Express and Expensify, and you think about spending money on a restaurant you have to get the receipt from the restaurant and pay a SAS fee for Expensify input, which probably happens at the end of the month.
you receive. and someone has to go through all of that and put it all together and just the fact that these are two different tools just really doesn't make any sense. Expensify doesn't necessarily have a context that you have swiped that credit card until they see the receipt. Amex doesn't have context on what the receipt is until you input it, so it's a very reactive way to run a business.
Ramp's rise has been because it becomes so easy to issue virtual credit cards. So we use Stripe Marqeta as issuing partner and can spin up credit cards much more proactively. So I'm just gonna give you an example of an engineering offsite that we did in Miami. We can use an HRIS integration, and figure out that every single engineer or person in the engineering org issue a credit card for $1,000 to be used only on flights and transportation.
All of that already has the accounting coding done. The card expires three days after the engineering offsite is over. So if you think about what enables sort a person who uses the card, it's a little easier. You don't have to do receipt matching. But if you think about what enables a finance team, it's actually tremendous.
And there are sort of two reasons why. The first is all of this spending is proactively approved, the maximum. that you can spend on this engineering offsite when you issue the cards. Second, all of the accounting, coding and contacts are encoded in the cart. You know that this is for an engineering offsite.
You don't have to go in and manually click. And then additionally, we have a lot of integrations, whether it's Gmail, whether it's Uber or Lyft, such that these receipts automatically flow in. So really I would say it's a card that allows you to be very proactive and add context to every single dollar that you spend.
I would say for a general employee, it saves you a little bit of money on receipt matching and counting coding, you don't have to do the thing where you chase down receipts at the end of the month. For a finance team and a CFO, it's a bit of a superpower.

Very interesting. Let's dive a little bit deeper into your team.
Can you explain how your data team is structured at RAMP and how have you scaled this team so far?

Absolutely. So day one for me at Ramp we were a four or five-person data team at a company that had already raised an 8 billion valuation, which is just nuts. Like it's it was the wrong size data team.
And so we certainly had a lot of growth. And I would say the answer right for me is it's like in perpetuity, assuming that how you have set up your team is incorrect or at least if it is correct for right the second, it will be incorrect in three months. So the number one answer I have is more on process than outcome, which is to create a culture of feedback where you understand where teams are being well supported, where they're not, where people are stretched too thin, where they're not stretched too thin.
 The ramp also really believes in tightly embedded teams of product engineering data and design. So that's how we're organized. The data team rolls up through the CTO and every single person on the data team is in one or two product and engineering pods. And so what this really means for me is when I hire someone, it is my responsibility to have an opinion on what the best product pod is.
I work obviously with the engineering design product stakeholders to do that, and then largely what I do is put together a 30, 60, or 90 for them. and the expectation is that 30 days into it, they're gonna rip that up, tell me why it's wrong, and tell me what they're gonna work on instead. So we really at ramp value having data team members embedded quite tightly into engineering and product.
We give them a ton of autonomy. It's very much like a bottoms-up process. And one other part about both our culture and how we organized is really what sort of I asked for from the team is that they come up with sort of their three top priorities for a product POD. Cross out the bottom two.
And that's really what we judge them on do they deliver the number one thing that they need it to in that quarter? So super generally, that is how our data team is organized.

Wow. And tell us a little bit more about these 30, 60, and 90 things. You said the first 30 days are about ripping apart whatever thing that is already there, what, 60 and 90 for?

Yeah. So I'll give you an example. I think in all jobs, in an ideal world, 50% of it, you come in knowing how to do very well and 50% of it you don't know how to do at all. And I think that's how you get a steep learning curve. That's how you inspire people. That's how you challenge people.
That's how you get people to feel like they're progressing. Certainly, for me, I think I came into Ramp with a lot of experience in growth and product analytics and very little in terms of enterprise sales and especially risk credit underwriting frog. And when I think about 30, 60,90's, I think we construct them in a pretty similar way where we try to set up a pretty tightly defined project that we know needs to be done and know needs some data team, essentially setting someone up for success on a project that's like a good thing to have in their 30, 60, 90. And then I'd say the other two things that I look for are a bigger murkier problem space with a team that they can embed in. So for example, working on price intelligence with this engineer and this product person going to these meetings, what network can we create?
What meetings can we put on your calendar? Who can you get to know? And then the last piece for me is my expectation of everyone on the data team also has the data team as a stakeholder. So really thinking about how can we have a team that raises the bar. How can we have a team that educates one another? How can we have a team that thinks it's its responsibility to accelerate everyone's career as data professional?
So largely I think that's the try to culture. I like to have and really in a 30, 60, 90 calling out, a project that someone's gonna be successful on, a team that they're gonna embed within a surface area where they can raise the bar for the data team.

And talking more about stakeholders.
How does the RAMP data team work with other teams like growth go to market your risk and compliance team? And one of the very common structures that we have seen across organizations is having a central data platform team that is responsible for the code data platform. And then you have those federated data engineers.
Who is supporting different functions? So how is that structure for you guys?

So I would say that really what we do is embed tightly into product engineering pods for every single part of the team. And that includes data platform. And so I'll give you an example of this, right? Like, many data companies, we are excited to introduce lower latency systems, and streaming Kafka feature stores.
all of these things I would consider to be data platforms the same way I would Fivetran the same way I would Airflow or dbt. When we introduce them, we do not introduce them generally, we introduce them for a very specific and bespoke use case with our heads up thinking about how it can generalize.
So I'll give you an example of this. When we rolled out some of our data science stacks for the first time, we didn't go across the entire company. survey all the data science projects and think about what to work on. We are tightly embedded into one team in the use case, which for us is risk and credit.
We said we are going to design a data platform and stack that works for the specific use case. And the data platform engineers that are going to work on this are going to show up to risk and credit sprint planning. They're gonna know a ton about how we underwrite. They're gonna get to know the engineers by name.
They're gonna get to know the product managers by name. They're gonna be incentivized on these KPIs. So largely I'd say for data platform stuff, the muscle that we've been working on is. If you can build something that solves 15 use cases, that's great, but let's be super precise about what the one use case you're gonna solve is first and work full speed to set up your data platform to solve that use case first.
Then figure out how it generalizes.

And glad you mentioned risk and security and compliance and credit. Unlike your previous experience with Wayfair or Drizly, while working in those data teams, the situation in Ramp would be probably different in the context of security and compliance.
When it comes to data, you're dealing with financial transactions, what changed from your data leadership perspective, in terms of the tools that you operate, in terms of the way you operate teams to be able to take care of security and compliance aspects of Ramp?

Yeah. So I don't think it's all that different from B2C and there are two different counterpoints, right? I think we have banking transaction data. We have KYC KYB bureau data. That stuff is highly sensitive. But if you think about Drizly or Wayfair, their customers are sort of individuals, right? So I think from a consumer protection standpoint, it's a little bit different.
 I think the number one thing I would say is the data team needs to lead from the front. and get people extremely excited cross-functionally about data compliance and raise that bar. Cuz it's not something you can do on your own. If you think about a company, right? You're gonna have whatever it is, a hundred, 200, 400 people that all have access to some level of privacy.
And if you think about compliance and privacy and culture, it is sort of just respect something that's built and drops and lost in buckets. For us, we really have to think about every individual. We have to think about every surface area vector. If you look at some of the stuff that the FTC has been doing in the US, they're prosecuting individuals as opposed to companies for the first time.
That is deliberate. That is a point that they're making. And so this needs to be something where you work cross-functionally with your engineering stakeholders and say, hey, Unlike other parts of tech debt across an org, this is not something we can pay down in two years. This is not something where we can focus on security and compliance two years from now, right before we go public or whatever the event is.
This is something that we need to invest in every single day. I have found that the only way to get privacy and security and compliance initiatives is by working on them a little bit every month. So said another way. You never really want to put it in a situation where you're what are we gonna work on this month?
The new product launch or private C security and compliance need to have a strong working group that is enabled by the CEO, enabled by the CTO, something that you invest in and every single month you say, what are going to be the initiatives that we push forward this month. And I think if you do that for two years, you end up in a pretty good place.

Yeah and does it change anything concerning the data engineering perspective, we are seeing a lot of data teams adopting to various tools like data catalogues or metadata management tools where the ultimate goal is broader collaboration across data assets.
Right? And this is pretty much contradictory to the goals you would have at Ramp in terms of data security and data compliance. So, What changes from an engineering perspective?

I have found is certainly that like water sort of running to the bottom, people will do their work in the area where they have the most access. And so if you think about data sets, oftentimes they're replicated in multiple places, right? You have them in your core. You might have something like a Databricks also pointed at your sort of like core SQL
you move that data via Fivetran to snowflake. You move that data in Looker, right? , I would say, right? Like we have user-defined Okta Group security access stuff for Snowflake, and we have it for Looker. We're really happy about that. But then the next question is how can you take that access level and have it permeate the entire org?
So set another way. If someone can't look up a row of data in Snowflake and Looker, they should not be able to do it in Retool or some sort of additional tool that's hooked up to a pro-SQL system. So this is something that we're working on right now is to think about if there is sensitive data in Snowflake, right?
Snowflake didn't generate that rope, it came from somewhere else. So how can we be proactive about moving upstream? Communicating that we have identified sensitive PI in our data warehouse, figuring out where else it went and stemming in at the source and making sure that whatever masking or data retention policies, we have as a data team.
also apply to the rest of the company. But fortunately, we have, some great privacy partners and engineering partners to work on this with us.

So you have had a couple of experiences in the past where you have taken completely legacy analytics infrastructure and kind of made an overall to the modern data stack.
How do you decide whether to build a solution in-house or purchase an existing tool that is out there in the market? How do you first of all make that decision?

Yeah. So a phrase that I love is. Build what is necessary and strategic, buy what is necessary and ignore everything else.
And so the thought there, and this comes from Zach Canners is really if you build something, it is not an asset. It is a liability. And that means it needs to be supported forever. And so really thinking about. What things that my team builds can I honestly believe will improve fastest within Ramps, four walls better than anywhere else?
And so I'll give you an example of things that aren't that right. The ability to move data out of Google Ads into Snowflake is not a core competency of Ramp. It is something Fivetran's working hard on every day with their engineering team. If I take that dependency on Fivetran. , I can assume that it will be more stable a year from now than it is today.
That is not the case. If I build that pipeline in-house, if I build that pipeline in-house and the data engineer leaves, I can assume that the pipeline will be less stable a year from now. So some other examples, right? If you're going to issue a credit card, take a dependency on striper, and Marqeta, they will be better at issuing credit cards than you will be a year from now, cuz that is their business model.
And I think largely right, that focuses you. highly on, sort of saying, what do I need to do better than any company in the world? And that's really what we try to emphasize building.

And assuming the case where you decide to buy an existing data tool, what's typically the process for you guys?
Because I think, so this question might be very helpful for a lot of data companies, and a lot of data startups who are looking to, provide solutions to companies at a scale of from. So what's your typical process? And by process, I would love to understand is how do you. Tools I'm sure unlike a lot of companies correct me if I'm wrong but you wouldn't have a standard procurement process in place, I think.
So it's still very much dream driven rather than procurement-process-driven. So how does that process look for you?

Yeah. so I will even start before we decide to buy the company or even try it out. One piece of advice that I have been given that's stuck with me is just the importance of asking for help.
All of these tools in the space move so quickly. There is no way. That I should know about all these tools. And that's true for everyone, right? And so I think sometimes my team thinks of me as someone who is extremely well-networked in the data industry. That is not true. My network is what I would call like just in time.
And I'll give you an example. We recently evaluated a tool called Metaflow. That's open-source data science. The sort of for-profit open core company is outer bounds. I joined their Slack channel. I DMed their CEO and I said, we want to know more about this tool. We have some questions. And he made time for me.
And I think that is one of the coolest things about whether it's data, Twitter, or these Slack channels. It's a very open community and you can ask for it. This specific guy Vile set us up with probably two or three people that have adopted Metaflow. We heard some phenomenal things about it.
We heard some things that are growing pains for it. We learned a ton about it. So I would say before even really hopping into kind of anything around procurement, that's where I start is just figuring out who can help me in the space, trying to get some time on their calendar and I think you'll be really surprised at who raises their hand and gives you great advice if you ask for it.

Oh, That's lovely advice. And so would it be fair for me to say that having a community around our data product, is a big kind of a consideration for you while considering these products

I think it is. And I think some of the best advice or context I've heard on this is a podcast with Jeremiah of prefect and Chetan of Benchmark on Invest, the best. They talk a lot about open-source and open-core products, especially for an open-source project.
It is exciting and it is terrifying if you work either as an investor or sort of an employee at that company, cuz everything you're doing is in public, and you can see if people are engaging in your slot community. You can see if people are downloading your stuff on GitHub or not. You can see if people are upgrading to the paid version.
So definitely, I think an engineering question that I quite like in an interview is how do you know which AWS services to use? Because I think pretty famously, right? They have over 200 services now, and some of them are, the building blocks of tech and others have quietly fallen by the wayside.
And I think it's similar, which is it makes more sense to take a dependency on something you think is going to be around two years from now and it's gonna be better tomorrow than it is today. And evaluating the community around a tool, I think helps you make those calls.

Right and in, one of your articles you mentioned this term called Layerinitis. And it was very interesting for me to see that term. Can you tell an audience what exactly is that and how you tackle it at the ramp?

Yeah. So this is a concept that is entirely copied from I might butcher the name, but Jean-Michel Lemieux. He had a great Twitter thread about it. Shopify VP. And the observation that he had is that by default teams put code where they can put it fastest as opposed to where it goes when considering the long-term effect on the overall system. I think unquestionably, this is something that I've seen at all stops on my data journey, which is people will use your work.
people will take your output. They will transform it for their specific use case. That is exciting. It is phenomenal that they're doing that. You should always have that ability, but you need to think about what that pull, like that gravitational pull towards centralization and building production-grade stuff is.
And so for us, right? Like many teams, we have dim fact tables, we have great modelling, we think about Kimball and then we have additional surface areas where people can say, I'm gonna iterate on this stuff. I'm gonna build things in my schema. I'm gonna be building things in notebooks. And actually, before you know it and this happened, we had Ramp have one of our first account takeover events.
And, we had people stay up late and they were doing incredible research on account takeovers, and it was all in notebooks and to a large extent. Ramp's entire account takeover fraud program for two weeks was running out of one analyst's notebook on their schema.
And it was like a bridge off of our repo and it had a ton of commits cuz they were iterating quickly. And all of that was incredible. And that's exactly what we wanna enable cuz fighting fraud is oftentimes like fighting fires. It's if you come up with a solution two weeks later like it doesn't matter, the house is already burnt down.
So we enabled that person to iterate quickly, but then thinking about all of this logic, everything that this person has built that will benefit many teams at Ramp is living in a pretty siloed area of both visibility and dependencies. We don't know what dependencies have been built on top of the data team, and so thinking about how can we invite that person into our code base, how can we show them how to commit?
How can we give them help on this journey? How can we take that work back and distribute it to the entire company in a version-controlled way? That is how you solve this Layerinitis problem. And the number one and two things that I think are important are, one, a culture of celebrating hardening code, right?
Because it is always easy to say, I would rather focus on the next thing. I wanna do the next analysis. I wanna build the next model. I wanna do the next product launch. It's harder to say we are going to take some of our work, refactor it, and build it the way it should be. So that's important. And then the other is a culture of really inviting people into codebases.
So I'd say this is probably the number one thing I've changed my mind on in the last year one year ago I would've said the dbt repo is for analytics engineers and analytics engineers only. That is a recipe for people putting business logic in other parts of the stack. So for me now, I think it's really about how can I invite people in.
how can I have them do work that adopts well to dbt? How can we teach them a little bit about dim and fact tables? How can we teach them about modelling? How can we think about making things accessible and looker for the entire company as opposed to in a super custom Databricks notebook with 250, 600 lines of code. So largely like those are the two ways we try to combat layerinitis as a celebration of hardening systems and inviting people into codebases and teaching them our best standards. Wow

Before we wrap up today's episode Ian, the last couple of questions before we let you go. What is that one, number one piece of advice that you would give individuals who are looking to build a career in data and analytics?

I think the number one piece of advice I would give. No one will ever stop you from learning something. And it is really surprising the extent to which you can become an expert on something in two weeks or potentially even one weekend of grinding on a new tool or package. So no one will stop you from doing this.
No one will tell you how to do this. But I think that the number one thing is just saying instead of wanting to know about Docker, watch some YouTube videos. Learn Docker. If you know how to build a Jupyter Notebook, you can go from building a Jupyter Notebook to building a flask app that is hosted as a surface at a weekend if you put your head down and grind on it.
So that's just the number one thing I'd say you can always read the documentation. No one will ever stop you. You can always figure something out quickly, but you have to invest in putting that time in.

Nice. And last question. How can the audience learn more about Ramp and the work that you guys are doing?
What would be the best way to keep updated with your work

I would say certainly Ramp's website, but then also Ramp's Engineering blog. We had a phenomenal post recently by Kevin. Who leads our data platform team for analytics engineering, talking about how Ramp saves money with some of our CI CD processes.
So certainly Ramp is all about saving companies time and money and hopefully, we can contribute back a little bit to the analytics engineering community as well. So I'd say start, yeah, start with the analytics engineering blog post on Ramp. Perfect.

So thank you so much, Ian, for your time for this episode.
I'm sure our audience would learn a lot and they would've received a lot of insights that would help them in their day-to-day work. So thank you, thank you very much for being a part of the show.

In this episode of Modern Data Show Gunnar Morling discussed his interest in software engineering and databases and his recent move to Decodable, a real-time stream processing platform based on Apache Flink. He talked about the importance of cohesive data pipelines, from source to sink, and how his work with Debezium led him to become interested in stream processing. Gunnar also discussed how Decodable provides managed stream processing based on Apache Flink, ingesting real-time data streams and processing them, and putting the data into other systems.

Senior Staff Software Engineer

S02 E03: Innovating the Modern Data Stack: Change Data Capture and Beyond with Gunnar Morling Senior Staff Software Engineer at Decodable

Hello everyone. Welcome to the Modern Data Show where we explore the world of the modern data stack with leading experts and innovators. Today we are thrilled to have Gunnar Morling with us. Gunnar is the founder of Debezium, an open-source project that provides a platform for change data capture to stream data changes from the database into event streams.
 He's currently working with Decodeable, a real-time stream processing platform based on Apache Flink and was previously a software engineer at RedHat. Gunnar has extensive experience in database technologies. Event-driven architecture data integration and has spoken at numerous conferences and meetups on these topics.
We're excited to have him join us today to share his experience and insights on change data capture and stream processing. So without further ado, let's welcome Gunnar to the show.

Awesome. Well, thank you so much for the nice intro. This was very nice. I'm very happy to be here. Can I correct one small detail?
I'm not the founder of Debezium credit where credit is due. It was Randall Hauch who created the project. I took it over after some time and I was the project lead, but I've not initially created it.

Gunnar Morling

Oh, thank you. Thank you so much for that clarification. So Gunnar, let's start with a very personal question for you. You have had an inspiring career so far, and the very first question I have for you is how did you first become interested in software engineering and databases, and what motivated you to pursue a career in this field?

Okay, wow. You are starting with the tough questions right away.
I like that. So to be honest, I think it partly was even a coincidence. When I was going to school, I was interested in all kinds of things; at some point I even was considering becoming a journalist, for instance. And actually, I'm very happy I did not become a journalist. But yeah, then I learned about computers. I always had an interest in programming. I started to program when I was still quite young and then I thought, okay, let me make a career out of it. Butthen, one step happened after another, right? So I explored some areas. I saw, okay, actually I'm interested in the data space. Let me go a bit there.
And one thing led to another, it was not real, like a super planned trajectory, I would say.

After spending kind of good amount of years at Red Hat and working on the Debezium project  you have recently moved to Decodable. Can you tell us a little bit more about Decodeable and what inspired you to join that company?

Oh yeah. So Decodable is a startup in the data streaming space. So essentially what they do, or, well, what we do, is managed stream processing based on Apache Flink. So this means, it ingests realtime data streams, let's say from Kafka or something like a Apache Pulsar or Kinesis processes them, and then puts the data into other systems..
So it allows you to do realtime data integration with all kinds of connectors using streaming queries. So you can filter your data. You can map your data, aggregate it, process it in time windows, and so on as it arrives. So it's not batch driven. You don't have to go to your data and pull for your latest data.
It all happens in a push-based manner in real-time. So that's what Decodable does in a nutshell. It's based, by the way, on Apache Flink. Now why I was curious about it is, well, I was working on Debezium, as you mentioned for quite some time, and after a while I, well, I realized, okay, Debezium and change data capture  – and I guess we will touch on it later on in more depth –  is a part of a data pipeline. This concerns itself with the question, how do we take data in real-time out of a database into a data streaming platform like Kafka? And I was curious. Okay, let me, I would like to explore the entire space. So I would like to explore how can we build cohesive pipelines from source to sink without having to integrate like, not quite fitting parts. How can we make this one unique unified experience? I was curious about this processing part of course, because I saw in the Debezium community the need for that. It came up all the time. So people had their change data capture streams, maybe in Kafka. Now they wanted to do stream processing. And I was again curious, okay, how can we use Flink for this?
How can we make a nice user experience based on all this stuff? So I was interested in this space. And I just felt, okay, this is something I would like to dive into in more depth. And then also, to be honest, I had been working always in large companies until this point.
So Red Hat, when I left, it was like 21 K employees, I believe. I had been working at even larger corporations before and I felt okay, I would like to have this small team, this small startup experience, like everyone on deck, everyone pulling into one direction. Everybody is super energetic and that's what I wanted to explore.
And well then I learned about the opportunity to join Decodable and I felt, okay, this is real what allows me to do this. It allows me to go into this small space, hopefully, make an impact on this small team, and that's why I'm here. And so far, I've been very happy.

Right. And what about stream processing makes it tough that captured or piqued your interest as an engineer? What makes stream processing even a hard thing? Why isn't a still a solved problem? What makes it tough?

Right? I mean, so I think at the core there is the stateful nature, which makes this hard. So let's say, you have two data streams and you would like to somehow correlate them.
Now, this means you need to reason about those data streams in terms of some sort of time window. And the question is how do you demarcate those time windows? How do you identify which data belongs together? How do you go about data which arrives late? Right? This happens very often.
You have some sort of time window-based processing and you really want to be sure you only can close or you only close a specific time window once you have all the data. And now what happens if maybe one hour later you still have an item for an earlier time window, which you already assumed to be fully processed?
So those kinds of things. Then, well there is no transactional semantics around it. In particular, when people come from a context of relational databases, they tend to think very strongly in terms of transactions. And I'm, maybe I'm doing some sort of data changes like I persist a purchase order.
This reflects in changes in multiple tables in a relational database, and only once this transaction has been committed, all those changes become visible to other transactions. Whereas with real-time stream processing, they don't have the notion of a transaction. I mean, at least not in commonly used systems, I would say.
Which means, establishing these transactional boundaries, that's an unsolved issue, I would say, and I'm looking forward to exploring that space and hopefully, we can provide some solutions there.

Yeah. And how to stream processing, like if you were to explain it to a very kind of budding engineer in terms of.
How does stream processing even work? Like how is it different from a database? In a sense, I had an opportunity to work with stream processing almost a decade back when I was working with this tool called Esper CEP. So that was my first exposure to this complex stream processing data.
Right. How does it even work? Like, let's talk from a fundamental perspective. So you have the stream of data that's coming in. Let's say, the most beautiful definition of stream processing that I've ever come across in my life is in a traditional database, you have data on which you run a query, and on stream processing, you have a query on which you run that data actually, right? How does that work?

I mean, the fundamental difference is in terms of this interaction, the direction of this interaction, right. In a relational database setting – or any kind of classic database setting, it could also be like MongoDB or whatever – this interaction is pull-based, right? So this means if you have a query and if you want to have new results, ou need to rerun this query and it will give you a fixed result set, which is valid at this point in time. And now in regards to how this query is executed, well, usually it runs in a pool-based way from the top to the bottom. So let's say you do something like select something from something. Right. Just to give you a very basic example, we would first go to this select clause. We would identify, okay, that's the stuff we wanna select. And then we would go to this from clause and we would fetch some maybe even one item from that fetch clause. And maybe then in a loop, we would process more and more.
Until our query is complete. So maybe we have like a limit clause where we would only fetch as many results as we need to have to satisfy this limit. Otherwise, we would do it until we have fetched all the results. Whereas exactly as you say, for a stream processing system, it's the other way around in those kinds of systems queries.
Evaluation happens in a pushed-based way. So this means in this simple example, we would start at the bottom. So we would essentially when there's a new row, which satisfies this particular clause, this would then push upwards to the select clause. The select clause would project whatever data it needs and push it to the consumer.
So it is, as you say, it's like a query you can subscribe to and you will be notified whenever there's a new result coming in

Right, so there would be a lot of state management internally, which for exactly manage the state of that query. Why talk, talk to us a little bit more about the state management.
How does that happen?

Yeah, I mean, so again, as you say those operators can be stateful. So let's say we want to do an aggregation, like, I don't know, the aggregated revenue of my purchase orders per category and maybe by day, something like that. So now this means if a new purchase order comes in, which would then, sit at the very bottom of this query, we need to identify essentially, which are those buckets in this grouping which need to be updated.
So we need to keep all the information. Or let's say we do some sort of join. Well, then we also need to essentially keep the state of this joint so we can add more rows to it; now, how is this implemented? Well, typically some sort of stay store is used underneath those systems. So let's say in the case of Kafka Streams and also in Flink you could use RocksDB as a state store, which then allows us to retrieve that state in an efficient way.

Right. And Apache Flink has been out there for a while, right? And you just mentioned like Decodable is a platform that is built on top of Apache Flink. So is that, is Decodeable like a kind of a managed version of Apache Flink, or is there something that you know, that's a common question that you often get, right? Is this a managed service? Is it just Flink packaged for enterprise suit applications or? Or is it different from that?
And if so, how is it different?

Yes. So it is a managed Flink in a sense, but then also with a special twist to it. And to take a step back there, In Apachelink, you have essentially multiple ways for how you can interact with Flink or how you can define your stream processing jobs with Flink. There are programmatic APIs, which you could use with Java or Scala, there's a Python API even. And then has also Flink SQL, which is a SQL layer based on top of Apache Flink. And actually, this is what we right now use in Decodable. So the user experience in Decodable is fully based on Flink SQL.
So this means you define your stream processing jobs in standard SQL, and then Decodable the platform takes care of taking that SQL definition and essentially, deploys Flink jobs in a cluster based on that. But that's the part you don't need to care about. So you solely care about defining the logic of your query.
What are the kinds of data transformations, joins, grouping, and whatever your use case requires. So that's what you think about. You connect it, of course with your source and sink connectors. So you can take data from something like Kafka and write it into, I don't know, Snowflake, let's say, or Elasticsearch.
But then how this all is executed, via Flink, this happens underneath. So in that sense, it's not a managed Flink service where you would have the Flink admin UI and you could use that. You interact with Decodeable via Flink SQL.

And do you also extend the Flink SQL in terms of like, introducing some kind of, a different kind of let's say event processing like a new rolling time window function or something like that?

Yes. Right. So there are a few specific additional UDFs (user-defined functions) which are provided there on top. What we right now don't have is the ability for the user to provide their own custom UDFs, that's a requirement which comes up now and then. We are exploring it, but it's not something which is there.
Right now, if a customer has a specific need and we feel this makes sense for the user base at large, we would add such a UDF and then everybody can use it. It's not something which you could do yourself. But again, it's an area which is an exploration.

Very interesting. And can you provide some examples of like realtime applications or use cases that Decodeable is well suited for?
Like any interesting or amazing use cases that you have come across your customers using Decodable for.

One of course, which I'm personally interested in is just everything related to CDC change data capture, and reacting to data changes in a database in real-time. So this definitely is a popular use case for Decodable. And so people then, for use this, for instance, to join multiple CDC streams and emit one joint aggregate structure to something like Elasticsearch. Because, you know what you wanna do in Elasticsearch, you don't want to you want… If you have like a data structure, which is like a purchase order – which is like a hierarchical structure of multiple items – you want to have this in a single document in your search index in Elasticsearch. And using Flink SQL and Decodable, for instance, allows you to join multiple table-level CDC streams into a single document. So that's a common use case. Then, of course, just filtering your data. Maybe you have sensitive columns in your database, which you don't want to expose to external consumers.
So you could mask those values, you could drop those values. Maybe you wanna establish some sort of data contract. Maybe there's like a schema change in your source database and you want to shield your consumers from that. You could use Decodable to establish a data contract, which gives a stable interface based on that raw stream. So that CDC is in the widest context, but then of course there are many other, let's say business use cases. Something like fraud detection. Maybe you have a stream of transaction data. Which comes in via Kinesis or Kafka. And you look at the location of where did those credit card transactions take place and maybe I see your credit card has been used in Germany, and like in the next minute it has been used in some store in the US, this probably is fishy, right?
So maybe your cart has been stolen or your number got revealed. Then we could do this kind of pattern matching, this kind of analysis or this real-time transaction stream. And then, I don't know, send, for instance, those suspicious transactions to another topic. There could be an application which consumes that and then raise an alert or send you a text message or something. Something like that. So we see that a lot. What else is there then? IOT I would say is also a big thing, right? So you have sensor data. I mean, I don't know, temperature measurements. or let's say here in Hamburg where I live in Germany, we have a network of bicycle counting stations. So whenever a cyclist comes by, you will be able to react to this event via MQTT. So you could take that data and, I don't know, visualize it on a map. So you have like a heat map of where are most cyclists and all this kind of stuff.

Wow. Very interesting. Very interesting. Talking a little bit about Debezium, right?
So can you provide a brief overview of what change data capture is and how it works and more importantly, why is that an important concept in the whole modern data architecture?

Right, right. Absolutely. Yeah. So what it is, I mean, as I mentioned, it's change data capture (CDC) which means it taps into the transaction log of your database and extracts changes from that.
So whenever there's an insert or an update or a delete the CDC process will react to this event which gets appended to transaction log in your database and it'll propagate this event to any downstream consumers. So just to take a step back there. All transactional databases have what’s called the transaction log, like the write-ahead log in Postgres, or the bin log in MySQL, the redo log in Oracle.
You always have that for transaction recovery, replication, and so on. And this is the canonical source of changes in a database. So whenever something changes, an event will be appended to the transaction log and log-based CDC, which is implemented by projects like Debezium, are a very powerful tool to react to those data changes.
And there are a few very important characteristics which come with this log-based approach. So for instance, we will never miss an event. Also if, updates or inserts or maybe an insert and the delete happen in very close proximity. Because sometimes people think we also could implement a query-based CDC approach. Where we go to our database and like every minute we poll for changed data. But then, well, if within one minute something gets inserted and something gets deleted, well you wouldn't even know about it, right? And you couldn't identify that something gets deleted, to begin with. And then of course you could say, okay, let me poll more often, like every second.
But it would create a huge load on your database and you still wouldn’t be quite sure that you don't miss anything. And all those problems go away with the log-based approach. So that's why I think log-based CDC, that's the way to go. And why is it important data architecture? Well, people, of course, have large volumes of data in their databases, and they would like to react to it with low latency.
And just to give you one very common use case, it's taking data into a data warehouse, something like Snowflake or Apache Pinot, maybe as a real-time analytics system. So you wanna do those analytics queries which you cannot do on your operational database, because it's not designed for that.
And now, of course, those analytical queries should work on current data, right? You wanna run your reports, you wanna run your real-time queries on fresh data, not on the data from yesterday. And this is why CDC is so important because it allows you to feed such a system like Snowflake or Pinot or Clickhouse or whatever it is, with very low latency.
So, for instance, I know some users in the Debezium community, who go from MySQL to Google BigQuery, and they have an end-to-end latency below two seconds. So within less than two seconds, their data will be updated in BigQuery and they can run very current queries there. And people realize that. And very often, what I also observed is, okay, so maybe users have one particular use case where they feel, okay, we would like to use CDC, we would like to have this low latency. And once they have done it, once they have seen, oh wow, I can have my data in two seconds, they want to have the same experience for other use cases. And they see, oh, I can also use it for, I don’t know for, streaming queries, for building audit logs, all this kind of stuff. And that's why people are excited about it.

Right. So is change data capture the underlying technology that supports the database replications as well?

So I would say, the technical foundation is the same, right? Because I mean, of course, there are different ways how replication works, but let's say something like logical replication in Postgres, it's exactly the same mechanism.
Whenever something gets appended to the WAL, the write-ahead log in Postgres, this will be sent over to any replicas. And in that sense, Debezium as a log-based CDC implementation is like just another replication client, only that then of course it takes the data, formats it into like a generic change event format, which is the same pretty much across all the Debezium connectors, and then sends it out to all kinds of systems, right?
It could be another database, but it also could be like a microservice or something like Elasticsearch, a cache and so on.

Right? So we're glad you mentioned this, generic change format. How does Debezium capture data changes from a variety of data systems, each database would have its different log formats.
How do you manage that and how do you even scale with that?

Yeah, that's a challenge. So, as you say, those formats and those APIs, how a connector would get data out of Postgres, out of MySQL, out of Oracle, SQL Server and so on. They're, they are different. And until the arrival of powerful open-source CDC tools like Debezium, this definitely was a challenge, because as a user you would have to explore all those ways for ingesting changes. And it is not exactly trivial depending on the specific database. So the good thing is, now with Debezium, which comes with support for a variety of connectors and databases, you as a user, you don't have to care about it, right? So the Debezium engineering team, they do this sort of original research, exploration and engineering, and then the goal is to provide one uniform change event. So for you as a user, then it doesn't matter Does this event come from… from which database does it come from? Right? Is it from Postgres? Is its from SQL Server? It pretty much looks the same. And by the way, the core structure of the events, of course, resembles your table structure, right?
So the schema, the core schema resembles the schema of your captured tables. And now what's interesting is that Debezium is becoming. And I, of course, I'm biased, but I would say it's becoming a defacto standard there. Because what we are seeing is that other companies and other teams, which are not even part of the Debezium core project itself, also use the Debezium change event format and the Debezium connector framework.
So, for instance, there is ScyllaDB, which is like an API-compatible implementation for Cassandra. And in general, just like a scalable NoSQL store. So they implemented their CDC connector for Kafka based on the Debezium framework. So now they also emit the same Debezium change event format. And again, it does matter then does this event come from ScyllaDB, or does it come from any of the built-in Debezium connectors.
It's the same for YugabyteDB; so they also have their CDC connector based on Debezium- And just lately, and I'm super excited about it,, Google contributed their CDC connector for Google Cloud Spanner to the Debezium project itself. So now this is part of Debezium. And again, you get change events out of Spanner in the Debezium format.
And I think this is just super powerful for users because, I mean, you always try to converge on a single database. But then it never is going to happen, right? Because you have different teams there, different needs, different requirements. Maybe you're doing an acquisition, then all odds are off anyways, and you always have the situation where you have multiple databases and one canonical format. That makes tons of sense.

Wow. Wow. And one important question that comes into mind is how does Debezium handle schema changes? What happens when the schema of the underlying table itself changes. How Debezium handles this?

Yeah. So the way it is handled is, essentially,  all the messages which Debezium emits, are associated with a schema.
How this looks like, this depends a little bit on how you have what kind of serialized format you are using for your message. So let's say you are using JSON, you could embed schema or you could embed schema into every message, which makes those messages self-contained. But of course, also, it's very verbose and generally has lots of overhead because if you think about it, you don't change the schema of your database tables like every day, right? So this means we will give you tons of messages with the same schema, which doesn't change. So, it's not what I would recommend in production, definitely people use what's called schema registries, and this means, now your message is they just contain the actual data, of course. And then just refer a reference like an ID to a schema, which is in the registry. Then this could be used with JSON but also, and I guess more often, this is used with Avro,  or maybe protocol buffers, Google protocol buffers.
Now, this means as a consumer, you just go to that registry. And you received a message, it tells you, okay, this message has, is schema 123. So you can go to this registry, get this schema 123, and I guess you would buffer it locally of course. And then you can interpret, let's say this Avro payload. Andthose schema registries, they come also with schema evolution rules.
So typically if there was a schema change, and we would push a new version, which let's say isn't backwards compatible. Maybe we drop a message, we drop a column, and then this new schema version would be rejected. So, that's one angle using a registry which enforces certain kinds of rules. What I also often see, and what I would recommend is, and this comes a little bit back to Decodable and Flink, to use stream processing to shield your users from certain kinds of changes. So let's say you rename a column. Well, in that case, what you could do is, you could use stream processing or maybe if you are just in Kafka and Kafka Connect, you could use a message transformation in Kafka Connect, and you could use it to add back that field using its old name.
So for some sort of transition period, you would have the same field, using the old and new names in your messages. And then this would allow consumers to react to that renaming at their own pace, and then you, I guess you would define some sort of SLA where you say, after six months or something you drophen finally the old name. Until then, everybody needs to be migrated. But stream pressing would allow you to, do this kind of… handle schema changes in a more graceful way.

Right. And so, so CDC sounds like a bit of a fairy tale as of now. It's like, it looks like a perfect system that has low overhead, very less performance bottleneck. There has to be a catch. So what are the cases where it doesn't work?

Well, I mean, I would be lying if I were saying that there is no operational complexity to all of that. I mean, it already starts with the database. You need to configure your database typically in a way, for instance, that the translation log is structured in a way so we can extract change out of it. So this means you need to, I dunno, reconfigure your RDS database, or you need to go to your on-prem DBA and tell them, Hey, can you change those particular settings in the configuration? So our database, for instance, keeps more extensive information in the logs so we can interpret them.
So that's the first part, one first challenge. Then another challenge is how do you react to failure scenarios, and what do you do then? So let's say, your connector crashes and you need to restart it. This means now, once you restart this connector, of course, you want to consume that change feed from the database from the point where you left off before.
And the way this works in Kafka connectors, there are offsets which are stored. And if we get restarted, we can go to this offset storage and we will know from where we need to continue to read. But then those offsets, they are stored in intervals in Kafka Connect. So it doesn't happen like for every record, which means if we crash, we don't get to commit an offset. Then we would read a few events another time, right? So we would have what's called at-least-once semantics. Oftentimes people wanna have exactly-once semantics. You can get this with Flink and Decodable by the way. But, that is a challenge. And then just to name one very particular example, in Postgres we use what's called a replication slot which is like the logical identifier of this connection.
And this keeps track of how far we have read the change stream. And now what happens or what can happen is, if you don't consume from such a replication slot, the database will not discard any WAL which comes after the latest offset, which has been confirmed for that slot. The reason is, well, if you want to continue to read from that replication slot at a particular offset, well, the database still needs to have the transaction logs for that particular offset.
Now, if you have this slot set up, but then you don't consume from it for one week, let's say, because I dunno you stopped that connector, and you forgot about it. Well, the transaction, the database will then keep, it'll retain all those transaction logs. And of course, this can be a problem: you might run out of disk space. So what I'm trying to say is you need to add some sort of monitoring to it, so you are aware of that situation. And that, by the way, again, is the reason why I joined Decodable, because there I feel we can provide such an experience for you.
So if we could make this a part of Decodable and give you this sort of alerting function, if you say, okay there's this connection which you configured, but you don't consume it, and now well, there's a chance that WAL piles up in Postgres, you should do something about.

Right? Right. And now as we inch closer to the end of the episode, one question I have for you is, How do you see the field of data management or stream processing evolving in the next few years?
I think so. And what impact do you think this will have on organizations and developers? We have seen a lot of fast-moving changes happen in the modern data stack in 2022. What's gonna happen now? What do you think is gonna happen now?

What I'm, one of the things I'm excited about is that we are embracing more and more that data gets duplicated and denormalized, right?
So, I mean, when I started it my career, it always was the aspiration, okay, this data sits on this database and there should be highly normalized and there should be no duplication. This is the one system of record. And  if you wanna have this data, you need to go to this particular database.
And now I feel, we are already more now, more and more to embrace, okay, data needs to live in different systems. It needs to live in our operational system and this realtime analytics store. So we can have like a realtime dashboard, this kind of stuff. Maybe we wanna push data at the edge.
So you, we haveread models of our data. I dunno, maybe in an embedded SQLite data. Close to the user so we can satisfy read requests with a very low latency because we have already the data close to the user. And I feel managing all this that's a challenge, right?
So you need to, of course, you need to keep this data in sync. So this is where CDC comes into the picture. This is where stream processing comes into the picture. But you need to think about, I dunno, who's the owner of specific data. Who are my consumers? How do they talk to each other? Who do we need to notify if we wanna do a schema change?
So there's lots of operational management, to all of that. And I feel there's a need for like, I don't know, maybe a cohesive platform which provides all of that. So I think that's a part which we're going to see. And then, yes, generally speaking, I feel like this trendtowards low latency and real-time data, this is just continuing.
So I feel people will move more and more to those kinds of streaming queries. Databases will have those CDC capabilities built in. I feel like in a few years it'll be odd if there was a database which wouldn't give you such an interface. So I feel, those trends definitely continue and also speed up.

Perfect. Perfect. So Gunnar, thank you so much again for your time on this episode. It was such a pleasure having you on the show and learning all of these amazing concepts from you. So thank you for giving your time.

Absolutely. Thank you so much for having me. And I don't know if people have questions for me, feel free to reach out to me on Twitter.
@gunnarmorling is my handle and I'm very happy to exchange and learn from you as well. Yeah.

In this episode of the Modern Data Show, Brennon York, Head of the Data Platform at Lyft, gives insights into the critical aspects of the data platform ecosystem in the early stages when there is no scale. Brennon also discusses the structure of the data platform team and new emerging technologies within the modern data stack that have impressed him, such as machine learning orchestration systems like SageMaker, Union-ai, and Flyte. The episode provides valuable insights into building a data platform that can scale with the growth of a company, enabling businesses to stay competitive in the fast-paced technological landscape.

Head of Data Platform at Lyft

S02 E02: Building for Scale: When and How to Invest in Data Platforms with Brennon York, Head of Data Platform at Lyft

Hello everyone. I'm excited to welcome Brennon York, the head of Data platform at Lyft to our podcast, the Modern Data Show. Brennon is a true expert in the field of data with years of experience under his belt, including his current role leading the data team at Lyft. He has also held leadership positions at other companies like Simpatica Medicine and Capital One, and even served as a board of Directors for the University of San Francisco Data Institute. Brennon's knowledge and skills include distributed computing data science, data analysis, and has a proven track record of creating innovative research prototype for businesses.

And not only he is a leader, but he also acts as a mentor, coach, architect, and technical lead, and has a leadership style that focus on humility, clear communication, and keeping the team motivated. Welcome to the show. Brennon.

Brennon York

So Brennon tell us something more about your journey. We would love to understand you started your career as a computer consultant and then, over switching over by multiple profiles over years today, leading up to your current position as head of data platform at Lyft. Would love to understand from you, how did that journey happen?

Yeah it's a funny story, I guess. I got my grad degree in computer security, actually. But even while I was in grad school for that stuff, I think what attracted me to the security realm was because you know, it was just it's very hard right?

Mathematically it's very intricate and tricky. That led me through a number of areas and that whole, while I found my niche in security, in distributed systems, So that's where kind of the start of, you know, this distributed system world comes in. That got me playing with Hadoop when Hadoop was coming out and it was all just map produce and things like that.

And I just fell in love. I always had a heart for the operating system and making things extremely performant and efficient. And then when you get into things like distributed systems to do map produce and you learn about lamport clocks and all that other fun stuff like it it's very challenging to a new degree.

And I just always had this, I don't know, love for making. 10,000 machines do one thing very well together, right? Data then comes after that. And, you learn about machine learning and, model training and all these things and what you do with these 10,000 or more machine, but that's where everything got started.

And then it snowballed. I just followed that interest and it started with security to secure those things. And then that moved into building analytical tools on top of map produce for other companies. And then it just kept on going and going until here I am. But I just followed the passion and yeah, that, that took me to Lyft.

Nice. And I'm sure all of our listeners knows what Lyft is and the kind of business you guys are into. But talking specifically about your role at Lyft , walk us through about your role at Lyft right now.

Yeah, so director of data platform basically means that, my teams and I, we work through managing all of the, I would argue, front of house and back of house for data.

So that means anybody who's using the app, there's events that fire and trigger those go through all the streaming, the online services that trigger, real-time ML, things like that, experimentation platforms that run through there. You know, then you get into S3. And you get storage, then you have the backend which is a lot of the traditional, your ETLs, your orchestration systems, your batch compute systems that then create, various reports or, model training, et cetera on the other end.

And we run basically all of that. At this point we look at everything as you know, trying to be the framework or the series of APIs that are for internal Lyft customers, right? To just move quickly with data. And just go from there.

Amazing. And you know, just so that our audience put things into perspective, what kind of data volumes are we talking about?

You know, it's funny. So I believe, I was checking the numbers recently and we compute on about a hundred petabytes of data, a day. I don't know if I can necessarily go into the event volume,

Because you can kinda extrapolate there. But let's just say, they're very high.

We're, I think the candid thing is we're not on the order of Facebook or anything like that, we aren't into exabyte scale. I would argue in petabyte scale. Which is a lot, right? That's still more enough. I think you know, one of the Lyft being completely in the cloud. One of our kind of companies that we look up to because I think there's a big difference of scale when you own your own compute and then scale when you don't, there are very different challenges when you deal with, on demand spot instance types, reserve instances, especially as I'm sure we're gonna talk about in a minute, as I, I feel like the market kind of moves to crunch all this stuff.

You know, I do think there are very different unique challenges there. And one of the companies that we look up to is Netflix, right? Like they, a long time ago, very cloud first, cloud native, and have paved the way for a lot of other companies to start looking at cloud as the sole kind of backbone for their compute.

And we have a great partnership with them and a number of other companies. But so they spearheaded a lot of it though, and you see them probably on the forefront of making more tools than most of the rest of us when it comes to like cloud tooling and things like that.

Yeah. And that brings to my next question in terms of like you say you guys love cloud, right? And as a head of data platform, while making your decisions in terms of building your own stack for various use cases or something like that, do you have an affinity to use any of the managed service provided by these cloud providers?

Or do you guys tend to wherever possible build stuff on your own and what's the rationale behind whatever your answer is?

Yeah, the classic build versus buy, I think whatever I say is probably gonna make somebody angry out there because it is very opinionated in some degrees, I think as an engineer, like I'd love to build. As you know, someone who has to represent the business I very much respect the buy and I think there are these varying trade-offs. I would say again, a lot of it as I'm still learning right in my own career, that the trade-off in my mind comes very much from the business need at the end of the day.

So there are a couple factors that come in, it's you know, what's the run rate? How much money do you have? Where can you put those resources, if you're gonna have someone actually build the thing? And then, like I think what's the long-term ROI there? 

Sometimes you might take a layered approach, where you would go with a managed service today and also build at the same time your own solution. Because long term, the maintainability sometimes of the long term solution, maybe a little bit higher, right? But your ability to mold with the business becomes so much faster than a managed service. 

So you've gotta weigh all those things in and I think there's a lot of times where admittedly, especially with a lot of these big cloud providers, I would say the managed services are great. You're probably not gonna get a lot of features necessarily as quickly as your business wants to move.

But if you need something simple you know, Kinesis or something for like Pub/Sub or whatever, if you just need Pub/Sub, right? And you can work within the limits of Kinesis, just use it. It's probably gonna be more cost effective now the moment you need, something else or more than just like the simple Pub/Sub.

You know, choose whatever you want. You know, you're getting into the Kafka territory for instance or something like that, and you wanna use all these custom features it's gonna be a while before you're gonna get those potentially out of Kinesis, right? You gotta go talk to Amazon and you know, it's a natural kind of evolution.

So for me, I tend to try to look at things like that and just say, where do we need to be with the business? Like at the bleeding edge and where do we just need to maintain? And I think when everyone looks at their data stack, you've gotta look at those things, right?

You've got Pub/Sub solutions, you've got orchestration solutions, you've got batch compute, you've got online compute, you could even bucket in just those four categories. And I'm gonna leave out machine learning right now, cuz I think that's a way more nascent territory but also has its own buckets as well.

And choose where you want to be at the edge and where you don't, right. We run our own Flink, we run our own Beam. As I think everyone could probably imagine. Lyft is a very online business. It's a very online company for those exact reasons. We need to make sure that we are with the business when we're online, batch compute and other things like that.

We actually do run open source but we don't need to be nearly as cutting edge, right? We can fast follow and use managed services and things like that.

Yeah 

You know, I know you probably won't be able to get into specifics but what do you think, what percentage of your systems are off the shelf managed services versus the one that you have built?

Roughly?

So open source, if it is hosted by, let's say a managed MongoDB by Mongo, I would say off the shelf.

I would say quick swag, we're probably 60 40. We host our own

Got it. Okay. Fair enough. Now the next question that I have is and this is something that, that there's a common fallacy that I think, so most of the engineering leaders fall into is building for a scale when the scale is not even there.

And you probably have experience across a wide spectrum of companies. You have seen the scale of Lyft. You probably have seen different-different scales. First question, what are the most important things within the broader ecosystem of data platform? What's the single most important thing that people need to think about in the early stages when there is no skill to be able to be ready or prepared when the skill comes in?

What are those core components where you would say spend as much as you can in the earliest days possible?

You know, it's a great question and I think in my personal opinion, the thing that matters most at every scale is the product, right? Data doesn't matter if you don't have a product or like if you're selling something, if it's B2B, B2C, whatever.

I would let that be the driver, the way I look at things nowadays, is scale when you need to. I think that's why these managed services are amazing because they work zero to one and one to a hundred. Maybe they don't work to a thousand or 10,000, you know, and I'm talking, when I talk scale, then its factors. 

I would argue at every factor you should just deal with the scale when it comes. You know, I think what was it, Knuth that basically said like premature optimization is like the worst thing you can do effectively. And arguably I think that is actually the pitfall that many people fall into is we think we can get ahead of the game and predict it. It's pretty rare, right? Even if, let's say you go hire a senior engineer who had done way more scale or whatever, and they come in and you're like, Hey, I want you to prep for X, Y, or Z. Well, what you still don't know is whether you're actually gonna hit X, Y, or Z scale. And you may end up wasting effectively those resources, right?

Everything comes down to money at the end of the day. And while it'd be great to be prepared for it, if it never comes that's probably a worse problem than if it does come and then you kind of need to put some band-aids on it. Or to the same point, there are a number of managed services that can handle the scale for what is a lot less price than to go off and run your own whatever, build your own X, Y, Z thing that is exponentially more, more cost-prohibitive, I think in the moment.

Like it's not gonna come out well on your balance sheet for many years. And again, so I think taking that risk is almost in my mind nowadays just not worth it. It's better to start with a bunch of managed services or a lot of these the vendor solutions and things.

I think they work great. Personally, I wish vendors and these other companies had a better relationship to know when their scale ended and you could say- hey it's not for us anymore, you know, and it unfortunately you have to go through that sad divorce which doesn't always work out super well.

But I think at the end of the day, that's, that in my mind is what I would advise everyone. Scale when you're there, right? Don't scale thinking you're gonna be there. I guess the other caveat I will say is for Lyft, we've done a lot of work in effectively simulating our ride volume.

So one thing that I can say is basically it's a giant performance stress test on all of our services, microservices, API data platform, et cetera. And I think, that allows us to handle peak loads when they happen. I think as you're looking towards scale and you need to be at least dynamic to handle these random peaks, those are the next kind of worlds that you would want to invest in. Then that'll usually shed light on areas in which you can go focus on.

Wow. Amazing. And just to understand a little bit more about the way you have structured your team and you know, so talk about this data platform team, like what does data platform team even does?

How is it different from let's say traditional teams like head of data, the teams under the head of data where you're more responsible around analytics and BI. How is data platforms different from analytics and BI functions?

Yeah. I would argue the big differentiator is that they're our customers still. So data engineers, scientists, analysts, all those, you know, and the tools that surround their work to some degree are for them to own.

We're kind a much more centralized function. You know, our job and ability is really to make sure that it is easy to build whatever you need. Tables, reports, models, those kinds of end state artifacts that are gonna power the business. You know, at the same time making it easy for say, production engineers to create a new event,

have that run through the pipeline, test it, et cetera. I think back to the question of like, how do we structure ourselves effectively, I look at the world as like online and offline with that storage layer in the middle that kind of links everything together which even before my time they had chosen, a file format and a structure and everything, which I think was probably one of the smartest decisions, right? We started with Parquet could gone with or right either or but we had Parquet and I think, having that good compressed column or format was like. The best ground.

At the end of the day, get your what people are calling the data lake, correct? And then you can build outwards from there, both online and offline. And so everything we do revolves around effectively reading and writing parquet you know, into our data lake. And so that's the first separation, right?

And then I think everything comes into admittedly around resourcing and things like that. In offline we have teams around, basically our core components. We have, you could argue it's effectively three teams. I could break it up into more let's just say it's some version of compute and query, right?

Then there's orchestration and then there is kind of the front end systems. We have some of our own, we have some vendor solutions, just easy ways to access the data, the logs, be able to diagnose what's going on. In the online world, we have the same thing. We have a couple, but I would argue like online front end systems as you're well aware of Amundsen, right?

That being one of them. We also have some internal things to do data quality checks like that. And then you get into our actual stream compute systems. And in between is that layer of persistence, right? These are the teams that leverage our stream compute systems to run these customized Flink, Beam jobs, et cetera to kind handle the volume of events and get those written in.

So is stream compute a major part of the work you guys are doing? I have heard Apache's Flink, Beam, Samza and everything. So is that a majority of, does it take a substantial chunk of the work that you guys are doing at your team?

Yeah, I would say, compute on both sides of the house is probably the most substantial so Flink Beam on one side Spark Trino on the other side and that is where the majority of our power goes into if those go down right, Lyft can't effectively do its job nearly as well. So that's it's our most critical point.

The next question is probably a bit of controversial one is, what are your thoughts on the modern data stack?

I believe as a data leader, you would probably have 10 emails in your inbox every single day where there is a vendor who is pitching the new category that they have created with this amazing tool and amazing technology. So my first question to you is, have you seen any implementation of any of these new tools and technologies that actually made a difference to you or your teams?

And if possible if you can take names, would love to hear those success stories of things that actually worked for you guys.

Yeah it's a great question. I think, my team if they are hearing this would hate the answer. And it's I don't think we've done enough exploring in this front to be honest.

And I think that admittedly gets into one of those pieces that we talked about in a prior question around like. I think there is a reality of the business need and the market, right? Like the market is compressing a lot of these tech companies lately. I'm sure we're all reading this stuff in the news.

And so I think naturally we're moving from growth to profit and I think as many people would want to say, oh, well, like that doesn't really affect like these lower layers, but it really does, right? Everything then becomes about efficiencies and things like that. What gets cut? Well, innovation effectively gets cut, you end up becoming like your innovations become running these things extremely reliably to a very low cost, you focus more on your reliability, your SRE style of work than you do around, I think Hoodie, Iceberg, Delta Lake, these kind of new upsert level tools and systems that I think are moving us in these way better directions.

But we personally don't get to necessarily play in that realm right now, like we have to just keep the business online. I don't think that doesn't mean that we can't innovate but like I said, I think candidly, that's actually where I get really excited like I said, my whole journey has been around, making things really efficient.

You know, once you start capping your number of resources, then all you get to do is be more efficient. I find that actually quite intriguing. But it's not like we're gonna be launching a new Amundsen anytime soon, whatever that new thing would be unfortunately, I think we're gonna make some incremental progress, effectively a portfolio approach of things.

But our portfolio for like long term endeavors is gonna be a lot smaller. So yeah, unfortunately I don't think we're gonna be there nearly as well. I will say my own controversial opinion on all this is you know, there were some papers that came out a long time ago by Google and everything else on streaming.

I generally align with the fundamental principle that like streaming is the basis for everything, and batch is a subset of streaming, and I know that's the Lambda Kappa architect. Everybody argues but my fundamental thought is that, and I think database technologies need to catch up in such a way that the event volumes that we're seeing in stream can be leveraged in effectively near real time. 

We need those reads and rights in those databases to go down along with the guarantees that we typically see and want immediate consistency, not eventual consistency and things like that, right? Normally you have that cap theorem and you've gotta trade off things. And so I think, that's what we wanna see, right?

We want all that stuff to get a lot faster that way you can do a lot of this, real time, online , computation things. We're starting to see that with like Druids, Clickhouses things like that, so we're getting there. I actually know a buddy of mine who's been working on this technology Pelosa for a long time, which is very cool technology, very nascent in this kind of database world which is all like bitmaps and things like that going into the same realm as Clickhouse and Druid. And I'm super excited to see where that goes. Cause I think all of those technologies are where we need to get to paired with streaming.

That's gonna be the way.

Nice. Amazing. And just on a very quick slide detour, have you ever explored any of the materialized databases, for example materialized.com or KSQL db, is that something that have you guys ever considered?

Okay. So next question what is that one?

You know, you talked about these, streaming databases and database that can adopt to changes other than this, has there been any kind of category or kind of any new emerging technology within the whole modern data stack that has impressed you and given the business constraints, given the business need all of their sorted and that is something that probably you would love to try it out someday?

Yeah, again I think admittedly, I've been so focused, I haven't had time to necessarily move out as much. You know, one of the things that I'd love to pull in would be the Iceberg Hoodie world. I know I mentioned that already, but I think that's also not super new, but I think that's an area that like very tactically I'd love to be able to try out.

Another area that I wish we could pull in is the Alluxio world. Of this kind of pinned memory, I still have this dream that like, it could actually help us speed up a ton of things, but we just, again its a resource constraint issue. And the last one is, just in general, like the area that I'm super passionate about or I really love seeing is, where a lot of the kind of machine learning world is going, we see a lot of this with SageMaker and Union-ai, Flyte which came outta Lyft. A lot of these like machine learning orchestration systems and things like that are moving forward. I think that whole world is really cool. And I think that's demonstrating the new needs, right? Like it, goes back to that streaming database thing, but also compute tied with caching, tied with versioning and kind of dynamic tasks and things like that. So it'll be interesting to see where all the machine learning AI world plays out in a lot of this.

Nice. And I also happen to notice that you have co-authored this book called Spark Big Data Cluster Computing in production. How did that happen?

Like everything in the world, it starts with an email and you say yes to something when you probably shouldn't have. And I will say it was great.

It was a super fun time. I am not an author. I'm not a great writer. I think, I'm sure the editor had a lot of work to do for me. You know, for anyone out there who's debating a book, I really, I would encourage you write 10 pages of something and then say whether you wanna write a book or not, because it was a great experience.

I don't know if I would necessarily do it again just because it's I think for certain people it's just a slog to get through, know, writing that many words, but hopefully people found it useful. I at a time when Spark was coming out, I think, I was there working with a lot of the Databricks, the early Databricks folks and things, and so had known a lot of things then now I'm definitely less well-versed

And you have also, you were academically involved as in adjunct faculty at university of Maryland. And do you see a gap in terms of what academia has to offer versus what the industry need. And if there is a gap, how do you think, young professionals or people who are just getting into in the industry can bridge that gap?

Yeah, so my time in Maryland admittedly, was also, I was actually teaching in grad course in computer security on system administration things like that. Then a lot more of what is probably more apropo is the time when I was on the board for in San Francisco, because I think that, that they asked a lot of the same questions, right?

How do we prep our young people who are coming out with the right skills and technologies and tools. And I think, at the end of the day, in my mind, nothing beats trying things out, prototyping. I think yes there's candidly a level of luck plus motivation and grit that gets everybody everywhere.

But I think, there's the reality of mostly all of the solutions that everybody runs and all of these companies is open source to some degree, right? The vendor solutions, most of them are open source, right? Confluent Kafka, right? Amundsen, Flyte obviously Spark, Databricks, et cetera, right?

There are these pairings or you know, they're out there, Starburst and Trino. But Trino runs at Facebook as well. I think it's easy to pull these things down, make some incremental improvements, right? That's a lot of where I got my start was pulling down spark, starting to make edits.

I made edits to the build scripts and things because I was like this is taking too long. And so I think just having a little bit of motivation just drive to just be passionate about something and follow the passion. I think that's the one thing I tell everybody, be passionate and go with it.

And so at the end of the day, I don't know if I would augment, I think every public curriculum is a little bit different. What I've focused more of my time on lately is coaching people who want to transition from other careers into a tech-based career, whether it be in data or something else.

And I think that's where, these questions again of do I go back for a four year degree? Do I go try to get a master's somewhere or do I go to one of these boot camps or something like that are becoming quite accredited. You know, in that domain, I see a lot of places where I think these boot camps and I think getting people's feet wet, shall we say, in like a nine month or a one year boot camp and things like, that's a lot of where I've been spending my time and focusing on now is again those career transitions and how to just augment all these other skills that I think we have.

I would argue that's the bigger need we're gonna people are gonna be passionate about computer science. They're gonna come out. It's the lawyers, the nurses, the doctors, the you know, the other marketing practitioners, recruiters, et cetera that all want to, start writing code and automating things for them. And I think that's a whole world of people that also have great ideas that we wanna engender.

Yeah. Nice. And you know, just before we wrap up our today's episode let me leave you off with one last question. Lyft has been a hot bird for innovation when it comes to data products.

You have, Amundsen coming out of it, you've mentioned Flyte coming out of it. Two kind of interlink question, what would be your advice to individuals who are working in big companies like Lyft working on some of the amazing cool projects and technologies to be able to incubate or create such projects within their organization and yet being able to spin off that project away from the organization and kind of share it with the community and eventually probably build a business out of it.

And B, what is it those these companies can do to be able to support that culture of innovation and kind of the amazing capabilities of those individuals to be able to contribute not just to the company, but to the rest of the world. What are your thoughts on that?

Yeah, no, great questions. I think to answer the first one, very candidly, I think focus on the business need is my advice. And I know that maybe not, again, that's not like what engineers necessarily want to hear, but I think if you build a product that solves a real need even a data product.

First you have to demonstrate that it solves a real need. And I think that is oftentimes where I see things end sooner than they should is because, it gets very pie in the sky and, oh, this thing could handle everything and do everything. And so my advice is build incrementally and build to a need.

Whatever that need is, right? That's the same thing for a startup, right? Build to a real need. And that is exactly why, for what we've seen Amundsen and Flyte like those are built for real needs that have existed inside Lyft. And that is why they came to fruition and they snowballed, right?

And they got to a point I think the second thing, which Lyft does really well is like you gotta let it go. To answer your second part of the question is I think, if I could give advice to any company, I think more often than not, while we think that it may be a competitive advantage, I think and there could be some argument for that as well. More often than not, I think, letting these technologies go and flourish often becomes more beneficial back to the inside of the company as well. Because, you know what happens is, let's say you let go this 10 person team who is building this thing and you let them open-source it they're gonna go and keep building on that thing because they're super passionate about it. And then, company X or whatever that let it go is then just gonna reap those benefits. And so you become an incubator of these things and I think we see this over and over again. There might be some companies who wanna hold some of these things in, but I think oftentimes, especially in tech many of us are quite altruistic and you know, we do see this, continue to come out. Again, and again and again. Twitter was prolific with a lot of, I think they started a lot of the kind of like open source streaming world with storm, and then it snowballed from there know, so I think we see this with a lot of the tech companies.

I don't think we're necessarily gonna see people holding back necessarily. But you know, you'll probably get features and components and I think that's okay.

That's a great perspective. That's a great perspective. So thank you so much for your time again, Brennon. It was such a pleasure having this conversation with you.

I'm sure our listeners would love.

Appreciate it. Thank you. Thank you so much for having me.

In this episode of the Modern Data Show, host Aayush Jain is joined by Kai Waehner, the Global Field CTO at Confluent, to discuss all things about Apache Kafka, Confluent, and event streaming. Confluent is a complete event streaming platform and fully managed Kafka service used by tech giants, modern internet startups, and traditional enterprises to build mission-critical scalable systems. During the podcast, Kai discusses the benefits of using Confluent over deploying Kafka, the role of a global Field CTO, and the company's complete data streaming platform.

A deep dive into the world of Data Streaming with Kai Waehner, Global Field CTO at Confluent

The Daily Life of a Field CTO

S02 E01: A deep dive into the world of Data Streaming with Kai Waehner, Global Field CTO at Confluent

Welcome to the Modern Data Show. On today's episode, we are joined by Kai Waehner the Global Field CTO at Confluent, and an expert on event streaming infrastructures. Confluent is a complete event streaming platform and a fully managed Kafka service, which is used by tech giants, modern internet startups, and traditional enterprises to build mission critical scalable systems.

He's also an evangelist, keynote speaker and a trusted advisor to customers and partners across the world. We're excited to have Kai on the show to discuss all things about Apache Kafka, confluent, and event streaming. Welcome to the show, Kai.

Yeah, thanks a lot. Great to be here. We have a lot of great topics to discuss today.

Kai Waehner

Okay, so let's jump into it. So first of all, could you tell us a little bit more about Confluent? Like is Confluent simply a managed version of Apache Kafka, or is there more to it than that?

Yeah, that's an awesome question. And actually, while. Confluent was founded by the inventor of Apache Kafka.

Obviously, it's much, much more today. When I joined Confluent around six years ago in the early stages there, it really was selling support for Kafka expertise. Today it's a really a complete platform, so it's much, much more than just using data streaming for messaging also, but really the whole ecosystem around integration, data governance, and then obviously also if they moved to the cloud in the last years.

The fully managed service is a key piece of that, but how we typical explain it really is that confluent is data streaming in a complete version everywhere and in a cloud native way. And this is the combination of what Confluent is doing and how it's also differentiating from just using open-source Kafka and managing that by yourself.

And what is the typical rational you know, that you have seen your customers having for using Confluent services versus going with their own deployment of Kafka? What's the typical rationale.

Yeah, there's a few of them.I mean, the number one, especially in the cloud, is simply that people have huge issues operating big data clusters.

That's not just true for Kafka, right? But also for our technologies like Spark and so on. And so when you use a fully managed service, which is really completely fully managed, and you can focus on the business logic or integration logic and just hand it over to the experts and that also includes, not just operations to reduce the risk and the cost, but also the 24 sevens apart.

Because most customers that come to us, they run operational critical systems that have to run 24 7 often for transactional workloads. And therefore, on the one side, it's truly about offloading the operations efforts and cost, but also reducing the risk. And then what I said in the beginning, like it's really.

Much more than just Kafka itself, but the whole ecosystem because as soon as you use Kafka for more than maybe interest into a data warehouse, then you have questions about data governance, about security, about encryption, and all these kind of questions around that, that we provide both from a tooling side, but then also of course, the expertise, because just the tools alone don't solve the business problems.

Right. Right. And you know, one another interesting thing that I wanted to talk to you about is your role as a global Field CTO at Confluent, right. And Field CTO is not a kind of a very popular term and I'm pretty sure that not a lot of people would be aware of the term Field CTO.

So can you help us understand what exactly is a Field CTO?

That's a fantastic question and I get it all the time. And so you are right today. The people don't know about that much, but it's coming up more and more as a new kind of shop profile, so it's not just a confluent, right? But you can look at many other software vendors in that space, like Snowflake or Databricks or Cloudera, VMware many more that are creating this role.

And what does it mean. It means in the end that really it's a customer facing role where you work together with a lot of customers and with that see their use cases, their architectures, but also help with their strategies and with their roadmap initiatives. That's the one side of the role. I talk to many customers.

In my case it's really global. Some other people only in a specific region. That depends also on the company. And with that expertise, then I can share the stories with other customers. But then also it's really about doing thought leadership. So if you take a look at my blog, for example, I'm sharing these case studies and lessons learned.

So not just about what's good, but also about challenges and trade offs and pros and cons, and sharing all of this with others, and others is not just our customers, but this is really thought leadership for the broader data streaming community in my case. Or if you go to Snowflake, they're talking more about data warehouse, for example.

But it's a similar story. And so it's really about this collaboration with customers, with partners, with internal teams. So that we all get better in data streaming because it's still the early stage for these cutting edge technologies. And so you people both internally in our company, like for new hires, but also our prospects and customers in the community needs education about what they can do of that.

And as last part, and in my kind of role of Field CTO, I'm also a public spokesperson. So in addition to just blogging or doing presentations, I'm really also an official spokesperson for the company for doing interviews or for example, I'm also working with research analysts like Gartner and Forrester to make sure that they also understand what data streaming is so that maybe we can get a new metric render for data streaming in the future.

And that's in summary what the Field CTO is doing and interestingly, because this question comes up so much right now while we are recording, I'm also writing a detailed blog post about this question and maybe we can link to that in the podcast then later, because it's really a super interesting question and also a very interesting job role.

Oh, absolutely. We would love to do that. And is solution consulting a part of the kind of overall profile or the purview of a Field CTO?

Yeah, so I mean, the difference is that what I'm doing really is not supporting the projects in deep dive right. So that's why we have consultants.

Or we also work with system integrators, with partners that do their projects. And we only provide the data streaming expertise. So not on that level, but in the end you need consulting on every level. And as a Field CTO in my role I solve one of the biggest challenges , which is that I can speak to both the business and the decision makers and to the technology people.

So, very often in my conversations and meetings with a customer there's the CIO level. . And there's also the elite architects, and there is the developers, and there is the business people that really need to solve their problems and all of them speak a different language and to get these nuances right.

This is the big challenge and this is also my capability as a Field CTO because I rarely go deep into things like Kafka or Confluent Cloud. Now, instead of what I do, I talk about case studies which are interesting for the customer. So when I talk to a retail customer. Then I share case studies from other retail customers that do omni-channel sales.

For example, when I talk to a bank and I explain them how another FinTech customer build a real-time trading app, for example, and this then is in a high level, this is where everyone in the company understands it, both the decision makers and business, but also the developers and then the specific teams.

I can do a follow up and go deeper or if we go even deeper, then I bring in other people because I'm not the deep expert that's our consultants and other people. And but on this level that's really where I do the consulting and engagements to help the customers understand where they can leverage the technologies we have.

And very important also where you shouldn't use them. This is also a very critical question and this is what you often don't find in the marketing materials of the software vendors, right? Because every vendor has the best tool and does everything right. But this is really also where people know that I'm trustworthy, right?

So I explain them not trustworthy can help, but also where we need other technologies and how it's complementary. And this is also what people really like when I come to them that they know they can trust me and I will only explain where to use it, but also where not to use it.

Right. And I will come to that point in a while in terms of when to not use Kafka.

But before I do that I have a question on How Confluent as an organization is supporting the broader Apache Kafka community? I know that, the founders of Confluent were the initial contributors to the Apache Kaka product, but as of now systematically. What are the initiatives you have within Confluent to support the broader kafka community?

Yeah, that's a great question. And for companies like Confluent this wouldn't work without the community, right? So, in the end, first of all our product, confluent Cloud is based on the same Apache Kafka that you use in open source and therefore What we contribute to Kafka is not just to our product, but also to the open source community, like the huge investment of ZooKeeper removal , for example.

This was a multi-year project where most of the staff was implemented by our team. I mean, it's super hard and critical because it's a hard operation at Kafka. And these kind of things are not just for compliment, but computed back to the community. So that's on the code level , and also, for example a few open source vendors have very strange strategies.

Like when they have a security fix, they only add it to their own product and then maybe half a year later to the community. So that's not what we are doing, right? So, and for that we also have a community addition, which you can use with an added features from us, but even if you use open source Kafka in the community, you can do that.

And that's a technology level. And in addition to that, Today the normal community stuff you are expecting from open source and from cloud services, and we are doing meetups worldwide. During the pandemic it was all virtual, but now it's back to on premise and data streaming with Kafka is really such a defecto standard in the meantime around the world.

So we are doing these meetups around the world to meet people. And this is really also, again, not just about confluent, this is about, and talks that interest everyone, like a Kafka meet is very technical. So we talk about the Kafka roadmap, about best practices and also we work together with partners, with customers, and even with competitors that speak there.

Like maybe then let's say. Redhead or Amazon, which also have a kafka offering. And this is the big contribution. And this is in the end also the business strategy because most of our customers already use Kafka and come to us, right? Because they learn about the advantages, not operating it by themselves, getting support, getting additional features.

So without a Kafka community we would be in trouble. So it's both a business strategy of course, but it's also the win-win for everyone because you can always also just use Apache Kafka. That's totally fine. That's the community adoption. So, only a small percentage of the Kafka community is then in the end adopting Confluent or another enterprise service.

It's getting more and more in the cloud. But still not everyone is using the cloud service, of course. So, and that's totally fine. And this is the great thing about such an open source community.

Right. So that's an impressive thing. So basically what we are saying is the Kafka that is there within confluent ecosystem versus the open source version is almost up to date.

Right? There is, they are almost always in sync. Is that a fair assumption? ,

It's even better than that for the community because in the cloud things changed a lot. So on premise with our confluent platform, it is exactly like that. So first, for example we the Cook data, the Kafka community releases Kafka, right?

Where we have 80% of the project commitment. But it's the community also of IBM and others. And then after it's released, we need another four weeks before we release our platform because we need to add additional integration tests to integrate in our platform. But the Kafka you find in our platform then is exactly the same, and it's more or less up to date. Only the integration tests need a few more weeks. And the cloud, it's even better than that because of the cloud and the cloud where you can do rolling upgrades. So they're re-op operated and we have the control, right. We don't have to wait for the customer. And there is even better because I'll give you the example of the ZooKeeper removal, which is a super hard thing.

And instead of just rolling it out so that the customers tried out in production, we are first trying out the zookeeper removal in our own confluent cloud services, first of all in development and then in some test clusters. But then we are also rolling this out in our cloud offering. The customer doesn't even see it because, and doesn't have to care because it's fully managed by us.

And only after we have battle tested in our cloud service where we have complete control only then we also hand this over back to the community where they know it's already battle tested. So if the cloud has really shifted a lot, and it's even better for the community because we battle tested first and then shift it to the community.

Wow. And tell us a little bit more about the zookeeper change. What was this all about? What was the motive and how will it impact people now?

Yeah, so, in the end, in the last, I would say 10 years, most distributed open source project used zookeeper as a key value store for meta data management.

That's had hadooop, that spark, that's Kafka, it's all these kind of technologies, right? But therefore the big problem is that. You don't have just one distributed system, but you have two distributed systems, one zookeeper cluster, which manages the metadata and then in the end it coordinates also the Kafka brokers, which is another distributed cluster.

And zookeeper was never built really for that scale and reliability you needed in Kafka. And so in most of the operations issues where you really had a P one downtime in many of these cases. Actually, it was not a problem of the Kafka brokers, but of zookeeper. And if you ask customers or have your own experience with operating Kafka clusters you will realize that most of the challenges are not operating Kafka, but zookeeper.

And so there are two big advantages of the zookeeper removal. Number one is the operations gets much easier because you only have one distributed cluster. You only have Kafka, which takes over the capabilities of zookeeper and does it in a much better way because it also uses the Kafka log, all these features from Kafka for it, which scales better and so on.

So, operation Simplicity is the one big benefit. And the second one is that it also scales much better. So a big limitation in Kafka in the past was that you can have only a specific number of partitions, right? Like, we typically recommend don't use more than a hundred thousand partitions in a Kafka cluster and something like maybe five to 10,000 per broker.

It depends on the deployment, but that's the basic rule of thumb. With ZooKeeper removal, now, you can also do millions of partitions with a single Kafka cluster because how it works under the hood now with just Kafka brokers it is simply optimized and improved and this is by the way, now the big game changer also to many of these other distributed streaming systems.

Like, a few years ago we had a few discussions about, hey shouldn't we use Kafka anymore? But maybe Apache Pulsar because what Pulsar was built for is really even more extreme scale. In theory because what they have built is not just two distributed system, but three of them with zookeeper, with Pulsar brokers and Apache bookkeepers so thats three systems.

And already two is super complex. To operate three is even harder. So we went the other way around and we architected in a way that in Kafka you only need one distributed cluster then, but can now still scale to millions of partitions. And this is much easier to operate. And also then even if you're using a cloud service, it's much easier because it needs less resources than three clusters.

And with that you have huge benefits no matter if you operate it or if you're just the end user.

Wow. That's amazing. Right? So one thing that always kind of captured my interest was around what happens after you have put in data in Kafka, right. In most of the cases right now, Kafka acts as a kind of let's say in the early stages of adoption, Kafka mostly acts as a message broker in most of the use cases. But slowly it then evolves into a complete full-fledged stream processing platform where you're not just only relaying messages from one application to the other one, but you're also processing that data and kind of taking actions on the top of that.

So first of all. Tell us, it's all there on the conference website, data in motion, right? So help people understand, what is data in motion? How is it different from data at rest? And why its tough that's the more important thing.

Yeah, so this is really crucial to understand and there's a lot of misconceptions and misunderstandings here, right?

So, therefore one of the first things I always explain is that Kafka is not a message broker. But before that even, it's really, as you mentioned, it's important to point out the difference between data at rest, which in the end means that you store data on a disc or in a database and you keep it there, and then you do a query to that with a SQL query with a web service or whatever.

So that's good for reporting or for training analytic model. But with data in motion, you can continuously act on data. While it's ingesting and I typically recommend to really start not from a technical discussion, but always start from the business problem, if that helps. And if you ask the business, whatever the business is, it really doesn't matter.

The business will always tell you that realtime data beats slow data. So if you ask the business if it's better to use data now And act on it, whatever the action is an alert, a payment, even just a report. But now it's better than later, no matter if later seconds later, minutes later or days later, right?

And with that's the game changer of data in motion with the streaming platform. That's the high level difference between where you still use a data warehouse for reporting is perfect for that. You still use a big data platform for training models, right? But for many use cases, Acting now is more business value, which can be reducing risk, increasing revenue, making customers happy.

That's depending on the use case, but this is the difference of using data in motion while it's happening. And with that, now spoken about the technology, this is the big difference to a message queue or message broker. A message broker is only here to send data from A to B that's great, right? But that does not add the business value.

The business value is when you also use the data in real time, and that's not what you do with a message broker. And therefore I explain Apache Kafka is four different components. Number one is the messaging component. That's what everybody understands and what people are using. However, number two, and that's really already the thing where people most underestimate is the storage of Kafka, because with Kafka, you also decouple the systems very well. You put it into the Kafka log, and then every single consumer can consume at its own pace because reality is that yes, realtime data beat slow data, but most systems so they are not realtime and some will never be real time.

And so you get data into Kafka once from a realtime messaging system, from a web service request response or from a batch workload. And then you have it in Kafka once, and then everybody can consume it one real time, one near real time, one request response, and one batch. They all decoupled because of the storage of Kafka.

And this is the biggest game changer compared to a message queue so that you really cannot just do messaging in real time, but also provide Data consistency across different systems because most systems are not realtime, and this is the biggest value of Kafka, right? Often even more important than the realtime capability.

And in addition to the messaging and storage combination, which is the core of Kafka, you even have Kafka Connect for data integration, and you have Kafka streams or KSQL for stream processing for correlating the data. And while you are absolutely right that most of our customers or end users with open source start with the billing ingestion pipeline, even there, even if you don't do stream processing, which is more advanced, but even in the beginning, you should always use Kafka Connect for that, right?

Because it's part of the Kafka ecosystem for doing integration with databases, with other message queues, with data warehouses that's built into the platform and you don't need yet another ETL tool or cloud service for that because even the integration and processing capabilities are built on Kafka, scalable, reliable, real time decoupling, guaranteed ordering.

All that's built into one platform. And this is really what makes Kafka so unique in the market. With a message broker, you need to add another ETL tool with another code base and infrastructure, and you need to add another storage system and you need another correlation engine with a Kafka ecosystem, you get all of that in one platform end to end that makes operations, scalability, and support much-much easier. And this is really why Kafka is so successful in the market.

Wow. Wow. That's very interesting. And you know, while the, all four of these components are very obvious and kind of, you have explained it in a very nice way there's one thing I would want to dive deeper into is the very last component of it that is stream processing.

And you talked about KSQL DB, right? So first of all help us understand how is KSQL DB different from, let's say flink or samza or beam? What's the fundamental difference there? .

Yeah, so let's first talk about what's the same, right? Because it's really important to clarify the stream processing in general means that you correlate data, right?

You get data from different systems not all in real time, right? So maybe one is a sensor or lock data, real time, high volume. And the other one is a database, an Oracle. And the third one is an ERP system like SAP with a web api and you get data in and out of these systems and stream processing.

Correlates the data. After you get an event in, and this can be stateless where you just take a look at one event, or this can be even state full events where you have sliding windows, like always monitory last 30 minutes or 30 seconds, and this is where stream processing is really, this is data in motion, right?

You act when an event is happening and this is the added value. No matter which screen processing framework you choose. And now, however the question comes up, which one should you choose for your project? Of course. And the short answer is that from a feature perspective, most of these frameworks at least the modern ones, like KSQL, Kafka Streams, flink maybe even spark streaming they overlap 70, 80% of their features.

So, all of them are pretty good and they have some nuances where they are better or worse. How I typically recommend customers to take a look at that is that in general, because this is all critical pipelines often at scale, right? The more components or infrastructures you add, the harder it is to operate this end-to-end with our data loss and the flow latency and also regarding cost and operations.

The less systems you have, the easier, and this is in my opinion, the biggest advantage of using kafka streams or KSQL Kafka streams as Java KSQL as SQL code. So that you can do stream processing just with one ecosystem because it's built on Kafka, right? You only have one distributed infrastructure in the end Flink or spark streaming that separate systems.

But they also have pros, right? Like. Flink is very strong in doing stream and batch processing. That's where Kafka is not the right tool in the end. And also Flink, for example, has Flink sql, which is ANSI sql. So it's really the same like you write against Oracle. That's not what KSQL is. And then Spark well, It's not real time and, but it's good if you already have spark in place or Databricks.

So all of them have that trade offs. But in summary, I would say evaluating in these categories. If you just want to do stream processing, think about do you need another technology or is just Kafka with Kafka streams okay sql enough for that then it's the easiest and cost efficient.

Otherwise, I think flink is really the other standard on the market. This is, the adoption is huge, right? And it's a great framework. So, this is the other framework we see vendors emerging around that cloud services are available. The open source community is growing like crazy. So that definitely a great option.

And then I mean I'm also, Databricks is doing more streaming these days. But Spark, I would say is mainly when you already have spark workloads and want to also combine that in the same cluster with some realtime data, then Spark is good. But these are typically the options. And then there is, again, products or cloud services around this, but I would say these are the standards in the market that you should take a look at.

And because , that's the same like we discussed in the beginning. This is where the communities are, right? This is where the adoption is. There's different competitors for that. And this is a win-win for the end user. And on, I would only take a look at other niche stream processing engines if for a specific problem that's built for that.

But in general, these are the right ones to take a look and evaluate.

Right. And in terms of KSQL DB how is it different from a materialized view? Right? Is that the underlying technology the same, like, the way you have materialized view in Postgres you have a few materialized database like materialized.com.

Is that a similar technology under the hood or is there something fundamentally different with the KSQL DB?

The fundamental difference is that we are here talking about data in motion. While something like a postgres is data at rest, that's the fundamental difference, right? So our viewers, in the end, you take a look at the data, what it is today, and again, in a database, it's a very normal behavior.

We know because we store data in our tables. Then we put some use on top of that, and then an end user can carry them with SQL or a web api. And that, but that's data at rest. With KSQL, you do exactly the same and the same with flink, right? You do the same, but you keep the state in the streaming application and then you can query it, for example, from a mobile app, rest API and carry the current state.

The fundamental difference is that the state is continuously updated in real time, and so when you do a query, you can make sure that you get the right information, or you can also turn it around and do push notifications after that change. Right? You shouldn't query all the time. And this is the big difference between stream processing.

You don't query for a change because that doesn't scale well. And you get notifications when something changes. And this is the fundamental difference. And here again it's not that one is better than the other. It's really depending on the use case. If you have a business intelligence tool like Tableau and just want to query data, then a database is probably perfect for that, right?

But if you want to get notifications in real time, like, when you use your right hailing app, right, in order a taxi, and this is not where you want to do requests all the time for millions of users, this doesn't scale. Instead when, an event is happening, like the taxi arrived at the location or when you did a payment and do fraud detection, then you continuously do that in real time.

This is where stream processing is built for and only stream processing scales well for that in real time. A database doesn't scale for that and it's not built for that.

Right. Absolutely. So talking about this coming back to the previous point that we talked about, so what are the typical cases where to not use Kafka?

I see that you have written a good post about it. Would love to for you to share it with the listeners as well in terms of when to not use Kafka.

Yeah. So that's very important. And that's also, I mean, I go to a customer or prospect, and this is one of the first things I discussed, right.

Because nobody's happy in a year if you find out too late that the technology is the wrong one. So use the right tool for the job and combine them. If a vendor tells you they can solve every problem, and some try to do that, it's not correct, right? Because it cannot work. And with Kafka it's pretty easy.

So, I would read, and that's what I did in my blog and presentation. When not to use Kafka, you can easily qualify it out in some use cases. The number one use cases Kafka is not for complex analytics, while Kafka is for realtime data and you can store it and you can have some simple query mechanisms like a key value query or replay historical data and guaranteed order for complex analytics like you do it with an NCSQL query with complex joints, that's not Kafka.

It's that easy. So for that, you use an Oracle database, MongoDB, a time series database. Use the right tool for the job. So it's not for doing complex analytics. It's complementary to these databases. That's why most people use Kafka Connect for getting data into Kafka in real time. And then each consumer can decide by themselves some consuming real time directly for alerting.

But some others need to get into a database for complex analytics, right? So that's the number one use case and the easiest one or the most important one. Besides that, there's other ones like Kafka often does not do the last mile integration. What does that mean? Last mile integration is, for example, when you have hundreds of thousands of cars on the street, right?

Like, the last mile here is typically MQTT. Maybe it is a web API if you are in the smartphone gaming sector. Or if you are for example in manufacturing to IOT and robotics, then it's also not the last mile integration, that's an IOT protocol then. So typically Kafka is for the server side, for the backend.

It can connect to clients and mobile apps, but not at scale. So the basic group of samples, as soon as you need to connect to thousands of client applications, it's not Kafka. So, for example, with Confluent and also our community division, we have a rest and HTTP proxy. So you can connect from a mobile at to Kafka, right?

With produce and consume, but not if it's thousands of connections. So that's the other thing to easily qualify it out. And the third one. Which also sometimes people try to solve the wrong problem if a technology is yes. We're talking about real time with Kafka, right? But always define what real time means.

It's that important. Real time with Kafka means that you can do end-to-end low latency around 10 milliseconds or slower. , which is good enough for 99 or 9% of use cases. But if you want to build a microsecond trading app like NestEgg, right? That's not Kafka and that's also not flink and not any competing technology that's not pulls up, that's nothing like that.

Or if you're doing hard, real time for safety critical operations, like, building the next engine for your plane or doing realtime robotic systems for human collaboration, that's hard realtime, that's sea or rust, right? So that's very different technologies. Kafka integrates with them to get the data out of there and correlate with SAP, for example.

But it's not for building safety critical systems. And so in summary, really Kafka is not for complex analytics. It's not for last mile integration if it's more than a thousand connections. And it's also not for hard, real time or microsecond latency. And that's in the end, an easy way to qualify out because that's not what Kafka or similar technologies are built for.

Wow, that was very amazing. That was very insightful in terms of when to not use Kafka. So Kai. Looking back on your career and your work with Confluent and event streaming technologies, what have been some of the most rewarding or memorable experiences for you?

Yeah. So as I said, so I'm now with Confluent for around six years.

So that was a time where most people didn't know Kafka at all. Today everybody's using Kafka around a hundred thousand organizations around the world. So it's really used everywhere became the defacto standard. Now it's really the question, how do you use it and how do you do advanced use cases? So I really see Two key innovations are things that are a game changer.

Number one, as we discussed already, is the cloud, where you use it really as a fully managed service so that you can focus on the business logic here. The critical pieces really to understand also that there's many cloud services for Kafka in the market. Many useful marketing but most of them are not really fully managed.

They still just provision the infrastructure and hand it over to you for supporting and that's not fully managed. So to really do the right evaluations and read the terms and conditions, but fully managed, and again, not just Kafka, but the whole ecosystem, that's really the game changer, including the connectors to other cloud services like Databricks or Snowflake or MongoDB Atlas or, so that's the game changer number one.

And then from a use case perspective, the real secret source in data streaming is the stream processing part. Most customers start with building pipelines because it's the easier part and still adds a lot of value. For example, in connecting you on premise applications to the cloud or connecting to your mobile apps and sensors and getting it into a database, super valuable and at any scale works with Kafka.

But then building use cases with stream processing like customer recommendations in real time while the customers in the web store, right? We all know the use case, like, you bought this item on Amazon. Maybe you take a look at that one, but that's more a batch use case. What customers are doing with stream processing is while the customers doing clicks you do a real time click stream analytics today instead of putting it all into Hadoop or Spark, running a batch workload and sending an email with a recommendation a day later.

That doesn't work because the customer already bought it somewhere else. But doing this kind of decision makings in real time with stream processing, that's the real game changer. And that's true for any industry retail, for upselling, for example, in the telco industry, for sending alerts to your customers about network outages and predictions, predictive maintenance and IOT fraud detection and payments.

Right? And that's what every industry needs because every industry has some kind of payments. And if you do fraud detection and batch in your data lake, you will still detect the fraud. But it's too late because the fraud already happened, right? You need to detect it in 10 milliseconds before the payment is accepted.

And so across industries, the secret source really is stream processing. You typically don't start with it because it's more advanced. But when you have built your pipeline first, then it's easy to add one more application or two that do the stream processing later. But that's really the innovations we see the cloud and then the stream processing.

Amazing. And, you know, kai before we kind of wrap up our today's episode, I have one last question for you. What are the challenges that you see are still persistent in the data in motion? Like what are some big unsolved problems that you see are still there in data in motion. And what are you guys working on to be able to solve that in the near future?

Yeah, so that, that's a great question. There is, again, two different kinds of answers. The number one problem and challenge is that people learned how to develop applications at rest. So, 90% or even more of the developers and architects on the market in the last 10, 20, 30 decades, they built a web service and a database, whatever technologies is or was.

So web services, rest APIs, doesn't matter, right? But it's an API or a SQL and that database, or it's not a data lake or Lakehouse, but it's data at rest and that's how you understand how to implement. Data in motion, and no matter if it's Kafka or Kafka Flink or anything else, it's a paradigm shift in how you develop applications because the patterns are different, the best practices are different, the technologies are different, the APIs are different.

So the number one challenge still today is education. People from the university, they get it right because they learn it already. That it's different. But for most people in the market education is the biggest challenges. And that's why we have on websites like our developer experience, right, where we train people about how to do that.

And of course, on top of that, we also build tools like we have a visual coding tool that you can drag and drop. Pipelines together like you did with your favorite ETL batch tool, but then it's streaming under the hood automatically and fully managed. So, that's the one thing to make it easier and to help people change this with this paradigm shift.

And then for the people which are already using Kafka, understood it for the first use cases. The real challenges here are really the more advanced once then, like, I'm talking about security, compliance and data governance. So these days, Many people are talking about the new concept of a data mesh right building independent data products, separation of concerns, domain-driven design, microservices. That's great to decouple everything. Each business unit can do their own things with their own technologies. And Kafka is the perfect data hub for that because it's real time scalable and decouples the things.

But with that is organizational challenges comes up. So who owns the data. How can you enforce that it's the right API contract? Who has access to the data? Do you have audit logging for that? How can you enforce encryption end to end? These are the questions that typically don't come up when you have your first pipeline from one database to another, but they come up as soon as more business units are using that and because this comes up.

So much in the last years and many of our advanced mature customers, they've built their own solutions on top. But because every customer now needs system, and this is really where we are solving this problem with products. On top of that, like if you take a look at Confluent Cloud today one of our biggest pillars, we invest into a data governance with the same things you actually know from your data lake or data warehouse, it's a data catalog. It's governance, it's data lineage, right? This, that's not new concepts. But they are mapped from data at rest in the past to data in motion where it's all real time and there is different challenges. But that's in the end how we help with education and with products to solve these problems.

That was very insightful, Kai. Thank you so much for that. So, Kai, as we kind of wrap up the episode for today, we again, thank you so much for being a part of this show. It was such a pleasure having you here and learning a lot of things from you. So thank you for your time.

Yeah, it was great to be here.

Thanks a lot.

Season 2