In the inaugural version of our Interview series - Amazing People in Data
 Meet Vladyslav: who started his career building software products, is an engineer at heart, and is
 now building data solutions in his role as CTO of <a class="mds-text-primary" href="https://www.mighty.digital/" target="_blank">Mighty Digital</a>. We had a candid chat with him on his journey from
 the software world into the data world, which is truly fascinating.
 Fun fact about Vladysav - He occasionally does some open-source development as a hobby and built a
 very cool tool called <a href="https://github.com/inlife/nexrender" style="color: #f66640;"
 target="_blank">nexrender</a>, for
 data-driven Adobe automation.
 
 In this super candid and fun conversation, we took a deep dive into his career trajectory and how he entered the data
 world. If you&rsquo;re building your data infrastructure, Vlad has some interesting advice for you!
 Vlad, tell us about your background. How and when did your journey to the data world start?
 I completed my higher education in software engineering and went on to work in the web technology field. And
 simultaneously, I was doing some game development as a hobby. At some point in time, while I was working actively on
 my first or second job, I got connected with one of our founders (at <a class="mds-text-primary" href="https://www.mighty.digital/" target="_blank">Mighty Digital</a>), and we kicked off our first
 startup. It was a cloth exchange service focused on women's fashion where I was doing primary web-related things and
 was responsible for most of the engineering and data collection. Data-world then was not as prevalent as it is right
 now. Mostly everything relied on some sort of general analytics tool. Eventually, our startup unfortunately failed,
 but we learned a lot from that experience. We went different ways, and then after about three or four years, we
 collaborated again on another startup. It was called Attractor and was more around data and analytics. So, my
 primary work and the start-up work were happening simultaneously. And that is how I started moving from software
 engineering to software engineering + data, where I built data solutions as a software engineer.
 Tell us a little bit about your work at <a class="mds-text-primary" href="https://www.mighty.digital/" target="_blank">Mighty Digital</a> and the kind of projects you&rsquo;re working
 on
 Most of the time, we work on product-oriented and consumer-facing projects, usually B2C apps. For example - one of
 our clients is a transportation app that allows people to travel on toll roads in the US. So there are paid roads,
 and you could either go all the way by attaching a sticker on your front window, or you could use the app and
 simplify your life. Another example is a scooter company that works internationally and manufactures their scooters
 and gives people an app for riding scooters. To sum up: we tend to work with startups and companies that are quickly
 growing and require help with that process, especially in data analytics and data infrastructure. Their growth
 shouldn&rsquo;t be bottlenecked by the data ecosystem that they have at the moment.
 There is a recent school of thought that is going around in the community around productisation of data. A
 lot of software engineering principles and development principles are being applied to that data domain. What
 are your thoughts on this?&nbsp;
 That's a great question. I think DBT is one of the primary examples of the new age shifts that started happening in
 the data field. Because before that, data was its closed field where you would have these data and analytics teams
 working in closed-off bubbles in a request-response fashion and only providing limited reports to other groups. With
 this approach, there is an interconnection between different worlds, i.e., data with software engineering. And as a
 software engineer, I think that's the greatest thing that could happen to data. This new mix of things improves
 everything because software engineering principles are strong. They have a whole history and were created for a
 reason. And if they are applied here, the situation will improve a lot.&nbsp;
 You recently posted your data stack on MDS.xyz. What was starting point of it and how it has evolved over
 time?
 I think this process is entirely organic. First, you look at the tools to analyze the data, like Amplitude, for
 example. Then you think about how to integrate this data collection into apps. For that, we can use something like
 <a href="https://www.moderndatastack.xyz/company/segment" style="color: #f66640;" target="_blank">Segment</a> or <a
 href="https://www.moderndatastack.xyz/company/mparticle" style="color: #f66640;" target="_blank">mParticle</a>.
 So these were
 the initial tools. After
 this basic setup, we start looking at much more advanced data integration, connecting multiple sources, etc. That is
 the point at which a data warehouse comes to play. So you would choose something like <a
 href="https://www.moderndatastack.xyz/company/google-bigquery" style="color: #f66640;"
 target="_blank">BigQuery</a>, <a href="https://www.moderndatastack.xyz/company/redshift" style="color: #f66640;"
 target="_blank">Redshift</a>, <a href="https://www.moderndatastack.xyz/company/snowflake"
 style="color: #f66640;" target="_blank">Snowflake</a>, or maybe even <a
 href="https://www.moderndatastack.xyz/company/firebolt" style="color: #f66640;" target="_blank">Firebolt</a>.
 And then, when it
 comes to data
 visualization, you can go with a pre-built data studio, <a href="https://www.moderndatastack.xyz/company/redash"
 style="color: #f66640;" target="_blank">&nbsp;Redash</a>, <a
 href="https://www.moderndatastack.xyz/company/looker" style="color: #f66640;" target="_blank">Looker</a>, <a
 href="https://www.moderndatastack.xyz/company/tableau" style="color: #f66640;" target="_blank">Tableau</a>, or
 something similar. At this point, you
 will start asking questions like, how do I enrich the data. You either go by building manual pipelines, where <a
 href="https://www.moderndatastack.xyz/company/meltano" style="color: #f66640;" target="_blank">Meltano</a> or <a
 href="https://www.moderndatastack.xyz/company/aws-glue" style="color: #f66640;" target="_blank">AWS glue</a> or
 something like
 that comes in, or you use
 <a href="https://www.moderndatastack.xyz/company/hevo-data" style="color: #f66640;" target="_blank">Hevo data</a> or
 <a href="https://www.moderndatastack.xyz/company/fivetran" style="color: #f66640;" target="_blank">Fivetran</a> for
 data
 acquisition. Then you ask how I
 model everything together to get a nice picture? For that, you get dbt. So it&rsquo;s a process that you discover
 over a while.
 
 How long did it take for you to build this whole architecture?
 It differs from project to project. For me, it was about a year or so. Not all projects require modeling, or not all
 projects require data acquisition. So, various things came at different times. Moving one step at a time is a more
 rewarding experience because you're building everything brick by brick.&nbsp;
 What key metrics do you track indicating that the data infrastructure you have built is delivering
 value?
 The primary metric for any data stack should be &ldquo;the trust&rdquo; in data. Because usually, whenever a project
 comes to us, people often say, &ldquo;oh, is the data accurate?&rdquo; And that's the primary problem that prevents
 data-oriented growth within companies. Because if people don't believe that data, they cannot make decisions based
 on it. The first thing we need to solve is trust, and we need to ensure that data quality is intact.&nbsp;
 We also are seeing an explosion of tools in the modern data stack. There are categories that are getting
 created every day. How do you feel about that?
 People are creative. New tools wouldn&rsquo;t get created if there were no need for them. There will always be new
 problems that we didn't see before, and new tools would also be required to solve them. Not every tool and
 technology will live for long, but people will use it if it solves a problem. And I think it&rsquo;s excellent!
 What would be your advice for companies who are very early in terms of their data maturity?&nbsp;
 
 It is a good question and the hard one at the same time. I think the best advice would be: to get someone who knows
 what they're doing. It is imperative. It does not matter if you want to hire an agency that will do everything for
 you or hire an in-house expert to build everything internally. You would need someone who has experience building
 systems like that. Otherwise, more often than not, you will lose a lot of time and resources. So, I would say
 getting the right expertise early on is very important.
 You have written somewhere that &ldquo;more and more businesses will rely on self-hosted solutions for data
 working and analysis&rdquo; Don't you think that would be a very big task for a small company in terms of the
 whole cost and building those engineering capabilities to maintain those systems?
 I think it's more about general trends. More and more data-oriented legislation, like GDPR or Ad regulations from
 companies like Apple or Google, are coming to play. So, we'll see a more considerable shift towards anonymization of
 data. Companies are worried about how and who is storing their data. And they'll be pushing more towards storing the
 data themselves. This way, they would have complete control of their data. For small businesses, I would start
 small, use something cheap and then scale up and migrate to something more advanced.
 What are the challenges about the current data space? If you were to start a data company, what problem would
 you be solving.
 We always look for new opportunities to build something incredible. Unfortunately, I haven't figured out something
 like that. Even though we have gathered many unique problems already, there are often more solutions to each
 situation. And ultimately, It's great because it saves you time.&nbsp;
 Rapid Fire Questions
 What is your favourite tool in the whole entire data engineering space?&nbsp;
 <a href="https://www.moderndatastack.xyz/company/dbt" style="color: #f66640;" target="_blank">dbt</a>
 What is a go-to place to learn about all things, data? Like what, any blog or newsletter publication that you
 read recently?
 <a href="https://towardsdatascience.com/" style="color: #f66640;" target="_blank">Towards Data Science</a> and <a
 style="color: #f66640;" href="https://letters.moderndatastack.xyz/" target="_blank">MDS Newsletters/Blog</a>.
 
 One thing that you like and hate about your job?
 Novelty. I love that there is always something new to learn and scope for growth. And the worst part of the data
 world would probably be data quality! ;)
 Very lastly you are based out of Ukraine. How have you been managing all of this and this entire case, how
 have you been holding it?
 We are truly surprised by the world's support, and how our defenders fight for our country is impressive. So, I'm
 genuinely amazed by everyone worldwide and the Ukrainian Armed Forces. What they're doing is a miracle, allowing us
 to continue working and helping our economy. And we're happy to do that. We're still in Ukraine. We're still here.
 So even though some like rockets fly in from time to time, it's okay. We are here, and we're going to continue to be
 here.
 You guys are heroes. I think history will remember you guys for a long, long, long time. There would be
 stories and fables written.
 Thank You!
 Check out Mighty digital stack&nbsp;<a class="mds-text-primary" href="https://www.moderndatastack.xyz/stacks/mighty-digital" target="_blank">here</a>

Meet Vladyslav: who started his career building software products, is an engineer at heart, and is now building data solutions in his role as CTO of Mighty Digital. We had a candid chat with him on his journey from the software world into the data world, which is truly fascinating

Building Modern Data Stack from scratch with Vladyslav

Meet Andrew, Chief Data Scientist at <a href="https://www.rasgoml.com/" target="_blank"
 style="color: #f66640;">RasgoML</a>, who went from solving complex math problems as
 Applied Math Professor to building machine learning models to solve real-life business problems. He is someone who
 has seen the data space evolve over the years very closely. In this interview, read about his super interesting
 journey from being a professor at Towson University to Chief Data Scientist at RasgoML and why he believes with the
 emerging times the boundary between data scientist and data engineer have diminished.
Tell us a bit about your background, Andrew
I've been in the data space for a long time. I graduated with a Ph.D. in systems and industrial engineering in the
 early 2000s, from the University of Arizona. This was well before data science even existed. Then I took a job as an
 applied math professor at a school in Maryland, Towson University and while working with some of the undergraduates
 I got involved in machine learning. I found that interesting, but also found business problems more interesting than
 the academic mathematics problems I was solving. In 2006 SAS Institute, the statistical software company contacted
 me and offered me a role in a relatively new group they were setting up. This was even before people were talking
 about data science and we were building neural network models that helped in identifying fraud in real-time on
 mainframes before any of the modern stacks really existed. I did that for a number of years and then went on looking
 for new challenges and different kinds of problems to solve. I then spent five years at <a target="_blank"
 href="https://www.moderndatastack.xyz/stacks/datarobot" style="color: #f66640;">DataRobot</a>, which is one of
 the largest vendors in
 the data science space. For the last two years, I was actually leading their go-to-market efforts in the
 entertainment space. Then in August 2021, Rasgo reached out to me with an offer to come over as their Chief Data
 Scientist.
How would you compare the role of an analytics/data engineer to that of a data scientist?
I've always been leery of that division because when I started, no division as such existed. If you think of me, I'm
 a &ldquo;full-stack data scientist&rdquo;. The reason why you're seeing divisions is because as a community we've
 gone out and we've told the data scientists that they need to focus on the modelling part of things rather than the
 engineering part of getting the right data at the right place. It&rsquo;s really hard to teach people to do that
 kind of engineering in academia and it tends to be ignored. So people come up with the skill sets to do data
 science, the machine learning part, but they don't have the software engineering skills to build robust production
 data pipelines. I see the data engineer or the analytics engineer coming along because a lot of data scientists just
 don't have the skillset to really understand how to build these robust pipelines. Also, it&rsquo;s an easy
 transition for software engineers who know how to build other systems to move into that analytics engineering. It's
 a much harder transition for those same engineers to move into a machine learning side of the things.
Let&rsquo;s talk about feature engineering- a term that you would only hear from companies like Netflix,
 Airbnb, Lyft, etc who have built complex machine learning models and systems.How would you explain feature
 engineering to someone working at a small company and what does it mean for them?&nbsp;
When most people talk about feature engineering it means simply converting the data into a format that machine
 learning models can handle. Let's take a step back and talk about the purpose of the machine learning model- it is
 to find signals in your data. So there's something you want to predict, some behavior you're looking for, and you're
 trying to identify that behavior and there's a lot of noise. So the machine learning models, try to filter the
 signal from the noise. As a data scientist, I see feature engineering helping that modeling process, and identifying
 signals by building things that help you identify interesting behaviors. This means feature engineering is more than
 just the conversion of data into features useable by the algorithms. It is the process of capturing insights from
 the business and the data that can allow the model to better identify the signal. This can be features capturing
 existing business knowledge or based on patterns the data scientist found in the data.
In addition, a lot of data has a time series component. In some sense, it's a history (e.g. customer history, a
 machine log history). Machine learning algorithms need a single flat file with one record per observation. This
 means the data scientist needs to flatten that history somehow. Fundamentally that flattening is feature
 engineering. This flattening can be as simple as calculate the mean or other statistics, up through complicated
 mathematics such as calculating the Fourier transform of the sequence.
Feature engineering in the modern data stack
Feature engineering in extreme low-latency use cases is a really hard problem and that&rsquo;s why <a target="_blank"
 href="https://www.moderndatastack.xyz/category/feature-store" style="color: #f66640;">feature stores</a> have
 been invented.
At Rasgo, one of the theories we have is the more feature engineering you can push into the database, the better off
 you will be. In fact, that's what a lot of data engineers do, they take Python feature code from Jupiter or Python
 scripts and rewrite it to SQL. Honestly, that's one of the scariest things for me because then I have to go validate
 that their SQL version matches what I did in my Python code. It&rsquo;s very time-consuming and difficult as there
 are many edge cases. I've seen mistakes made all the time. That's why, if you can do it in SQL, I think you're
 better off, but I'm not a great SQL programmer.
A lot of data scientists aren't great SQL programmers, and honestly, SQL makes some things much harder than it needs
 to be. At least from a data science perspective. But the advantage of going to that SQL environment is now I can get
 to near real-time that as data comes in, especially with tools like Snowflake which can process data really fast I
 can actually do relatively complicated things in a hundred-millisecond timescale.
Web latency is not an issue anymore. Back when I was building fraud models in 2006, where contractually the bank had
 to make a decision in a hundred milliseconds and the fraud model was a small part of that. There's a lot more time
 was spent on decisions like: Do they actually have the money? Is there a credit risk? Is everything technically
 correct? And the fraud models were trying to be one-millisecond latency .
We were doing complicated feature engineering, trying to figure out how do I do this in-memory processing where I
 don't have to move data because moving data is expensive.
At high latency, you can do it as you're doing it. Although there are benefits of moving it into a SQL engine, with
 high enough latency, pulling the data into a different system, processing it and scoring it will still work. And
 this approach can be useful as it avoids the testing that needs to be done if the code is rewritten. At moderate and
 near real-time latency, you have to move the feature engineering into the database. This is where where the modern
 data stack really shines.
At low latency, this is where tools like <a target="_blank" href="https://www.moderndatastack.xyz/company/tecton"
 style="color: #f66640;">Tecton</a> comes
 into picture.
What is the most challenging part of building and scaling a data team?
It's really hard to hire data scientists because while it&rsquo;s relatively easy to identify technical skills what I
 found the most successful people in my data team are problem solvers more than they are technologists, and it can be
 really hard to find one. From my perspective, when I try to hire, I don't care about technology skill sets. As long
 as the individual has experience in a related technology, they can learn the skills we need quickly enough.
The other problem that I run across when hiring data scientists is they want to build models and that's a small part
 of our job. As a data scientist, you can't be successful without understanding what the input data is. It's your
 model, you have to learn about the political pain of change management in the end. You have to be able to explain or
 be involved in the explanation. As a leader, that's partially my job, but you have to be involved in that
 explanation to business leaders, to help them understand how to use this, what it means. The hardest part is
 managing business expectations, especially for junior- you're going to have to do a lot of just kind of low-level
 data work and you have to be happy doing that because it just needs to be done as part of your job.&nbsp;
What do you think are some problems in entire data lifecycle from production to consumption that are still
 unsolved?
I first want to highlight the amazingness of the change to ELT from ETL because before, I was stuck with data that
 had been transformed for other purposes and it wasn't quite right for what I wanted to do. So when Fivetran's and
 Airbytes of the world said that- &ldquo;Hey, we'll just load all of that data into Snowflake for you&rdquo; I loved
 that.
Currently, I still think the data catalog problem is unsolved. I've just never seen a data catalog work. Maybe
 someone comes up with a silver bullet, but I think it's more likely to be political and business decisions, not
 technological.&nbsp;
RAPID-FIRE QUESTIONS
What is that one tool or platform that you think have really changed the life of data engineers or analytics,
 engineers, or data scientists and why?&nbsp;
I think it is <a target="_blank" href="https://www.moderndatastack.xyz/company/snowflake"
 style="color: #f66640;">Snowflake</a>. Snowflake just blows me
 away. I got introduced to snowflake and that's one of those moment- Wait! I couldn't even process this data before
 and you just gave me an answer! It's just amazing. And then the way they're growing is just tremendous.&nbsp;
What is your go-to place to learn all things about data, like any blogs or any publications that you
 regularly follow visually?&nbsp;
I follow a number of them. Like- <a target="_blank" href="https://towardsdatascience.com/"
 style="color: #f66640;">Towards Data Science</a>,<a target="_blank" href="https://www.kdnuggets.com/" style="color: #f66640;">
 KDnuggets</a>, <a target="_blank" href="https://www.datasciencecentral.com/" style="color: #f66640;">data science
 central</a>. I don't find one that's consistently better than any of the others.&nbsp;
What is the one thing that you love about your job? And one thing that you just hate about your
 job.&nbsp;
I love solving problems. I love seeing the value I've produced. The hardest part, the politics that go into that, we
 still run into the situation where someone else owns the data. And I have to beg for the data. Neither of these are
 that enjoyable, but they are necessary parts of the job.

Meet Andrew, who went from solving complex math problems as Applied Math Professor to building machine learning models to solve real-life business problems. He is someone who has seen the data space evolve over the years very closely. In this interview, read about his super interesting journey from being a professor at Towson University to Chief Data Scientist at RasgoML and why he believes with the emerging times the boundary between data scientist and data engineer have diminished.

On being a full stack data scientist

Meet Dunith: a developer advocate @ Startree, whose blogs you might have come across on topics
 ranging from analytics, streaming processes, and modern data stack in general. In this interview, you&rsquo;ll learn
 about some of the awesome things he&rsquo;s working on @ Startree, unbundling of the streaming stack, his advice to
 the budding writers, and his data journey so far!
 Tell us about yourself, Dunith
 I graduated in 2010 and worked as a Java developer for around two years. Then I joined WSO2, which is a Sri Lankan
 open-source middleware company. I got into their big data team and was one of the key members t who drove the data
 analytics product vision. Then over a period of time, I eventually entered into consulting and ended up as a
 solutions architect at WSO2. After that, I started writing about my data experience, and my blogs started gaining
 good traction on the internet. I worked there for around 8 years and was eventually discovered by Startree and
 joined them as a Developer Advocate. Currently, I am working closely with the Apache Pinot developer community,
 helping people to adopt Pinot, educating people about it, and creating content about not only Pinot, but also
 analytics as a whole, especially modern data stack, stream processing, and streaming databases.
 Can you tell us more about Apache Pinot and the core hypothesis behind it?
 Before answering that question, let me explain two things
 On one hand, we have traditional data warehouses like Big Query, and Redshift, and these are OLAP data warehouses. On
 the other hand, we have stream processes like Apache Flink.&nbsp;
 Pinot sits in between these two technologies.
 By definition, Pinot is a real-time OLAP database, or a streaming database. You can think of Pinot as a black box
 that can ingest incoming events from a streaming data source and make them ready for querying in a matter of
 seconds. Also, Pinot allows you to query the ingested data with low-latency and high throughput.&nbsp;
 How exactly is Pinot different from ksqlDB or any other materialised databases in terms of the basic
 architecture?
 In Pinot, we write data into columnar formatted files called &ldquo;Segments&rdquo; and they are scattered across
 multiple nodes. We query these segments through the &ldquo;broker&rdquo; that knows where to find them and propagate
 the query in a fast and efficient manner. Queries are executed at the node level and results are
 &ldquo;gathered&rdquo; by the broker and sent back to the client. In short, Pinot queries work on
 &ldquo;scatter-gather&rdquo; principle. Conversely, ksqlDB and Materialize work on the principle of
 incrementally updated materialized views where new data is constantly passed through them.
 Can you share with us an example of a company using Apache Pinot for a specific use case?
 A very popular example and easily understandable is LinkedIn. On LinkedIn, you will get a notification when someone
 views your profile in real-time. You will see the recent people who have seen your profile within a dashboard, this
 dashboard is powered by Pinot.
 Another example is the merchant dashboard provided by Uber Eats. That dashboard allows restaurant owners to see
 real-time metrics about their orders, including current order count, total revenue, trending menu items, and
 customer satisfaction metrics.
 In one of your articles you mentioned that &ldquo;there needs to be an unbundling of the streaming
 stack&rdquo;. What would that unbundling look like?
 There&rsquo;s a misconception in the streaming space that a single vendor platform can solve all the problems of
 streaming analytics. Surely it looks like a bold statement which can&rsquo;t be realistic anymore. A
 streaming analytics platform (or a stack) consists of multiple components that play a specific role in deriving
 insights from streaming data. We need to identify these components and their role before planning out a new
 streaming stack or converting an existing batch analytics system. The article you mentioned is an attempt
 to discover typical elements, their role, and potential vendors in the space to implement them.
 In summary, we can &ldquo;unbundle&rdquo; or &ldquo;decompose&rdquo; a streaming stack into a few critical components
 as follows.
 <ol>
 <li aria-level="1">A streaming data platform.</li>
 <li aria-level="1">Stream processing applications (stream processors)</li>
 <li aria-level="1">A place to store historical data (data lake)</li>
 <li aria-level="1">A serving layer</li>
 </ol>
 What would be your advice for someone who would want to write content in and around data?
 Good writers are not born. You can become one by mastering the art of transforming what&rsquo;s inside your mind into
 a flow of words. I&rsquo;m still trying to master it.
 Writing is about expressing your thought process in mind into words so that someone can easily understand it. This is
 how I define writing. So before writing anything you have to first identify what you are trying to write.
 If you are a beginner, I would suggest you start writing about one of the recent projects. Think from a point of view
 of a reader, how this information would be useful for my audience if they saw this. One should always define their
 target audience. For whom they are writing for.
 There should be a good connection between your first word and the last word. Write at least 100 or 200 words on a
 daily basis. Keep exercising for your brain.
 What problems do you think are still unsolved in the modern data stack ecosystem?
 I think we are yet to improve the data reliability, discoverability, and governance in the streaming data space. The
 current technologies in the market are capable of getting you faster data. But you still have to make sure that the
 data is accurate and reliable. Also, as your streaming platform evolves, you need to pay attention to
 data governance, things like how to share, enforce, and evolve event schemas. Also, we need the ability of tracing
 event lineage-observing an event from its origin to the destination over time.
 RAPID-FIRE
 What is that one tool or platform that you think has really changed the whole modern data stack?
 I think it started from redshift as I read. Redshift has changed the way that people think about data warehouses, and
 then it did the first data cloud data warehousing and opened the platform for others to build.
 What is your go-to place for learning about all things, data, any particular book or newsletter or
 publication that you would recommend?
 I follow data influencers like Ben Stancil, Gunnar Morling, and several newsletters including Data Engineering
 Weekly. Also, I like to read engineering blogs of companies like LinkedIn, Uber, and Netflix because they put out
 all the new stuff.
 One thing that you love about your job and one thing that you hate about your job
 The best part about being a developer advocate is educating people. Teaching is something that comes naturally to me.
 I love creating content that caters to different audiences.
 There&rsquo;s no such thing as hate, but yes I definitely get tired sometimes because it takes a lot of energy to
 learn something, digest it, and make it educational for the rest of the world.

Meet Dunith: a developer advocate @ Startree, whose blogs you might have come across on topics ranging from analytics, streaming processes, and modern data stack in general. In this interview, you’ll learn about some of the awesome things he’s working on @ Startree, unbundling of the streaming stack,

Unbundling streaming stack with Dunith

Kelly served the U.S. Air force as a Linguistic Instructor and later took a turn in her career and went on to build data teams across different organisation.

Unsolved Problems of Modern Data Stack with Kelly

Meet Juan: who turned his Ph.D. research into a successful company, Capsenta, and now is a Principal Data Scientist at data.world. He believes data management needs a paradigm shift and that shift is including social aspect to the business. In this interview we touched base on his journey, how the data scientist role has changed over time, the ABC of building data products, and the socio-technical approach to data. Read about his journey and his work which are amazing as his podcast. Tell us about how you went from being a web developer to now a principal scientist at data.world. My parents are Colombian, but I was born and raised in the Bay Area, in San Jose. When I was 10 years old, I moved to Colombia and that&rsquo;s where I started my undergraduate education. In 2004, a guest professor, Oscar Corcho, came to give a seminar on Semantic Web, and I was fascinated by the topic - what&rsquo;s the next generation of the web or how can the web be smarter. That&rsquo;s what caught my attention. After that, I transferred and moved to Austin to finish my undergraduate degree at the University of Texas.&nbsp; I've always been interested in research, but at the same time, I always liked entrepreneurship. In 2005, I started my first company. I had an engineering base in Colombia, my business partners were in Switzerland, and I was in Austin. Later in 2006, I met Professor Dan Miranker who asked me: What is the relationship between relational databases and semantic web technology? And that question changed my life. I really got interested in that problem. So I applied for a PhD at the University of Texas, got accepted, and left the startup.&nbsp; The thought behind the application of the research was: how are semantic technologies going to take off? People have their data in so many different relational databases like Oracle, SQL server, etc. How are they going to put all this together? Well, we had an answer from our research. There was a startup opportunity there, and that&rsquo;s what we did. In 2014, I officially started the company Capsenta with my PhD advisor, Dan Miranker. Capsenta was doing semantic data virtualization. Basically having your semantic layer, and mapping that back to all your different databases. This was before dbt was a thing. data.world was actually one of our customers and one of the founders of data.world, Bryon Jacob, did the tech due diligence for one of my investors. So I knew him very early on and I'm like, &ldquo;who is this guy asking the really hard and really smart questions?&rdquo; We really hit it off and just kept talking for a long time, sharing ideas. Then it came to a point when we decided we should be working together. Both companies were a match in terms of the mission, the vision, and the technology. In 2019, Capsenta was acquired by data.world. That&rsquo;s how I ended up here as a Principal Scientist.&nbsp; In one of your blogs, you wrote that &ldquo;data management can&rsquo;t just be about technology. Data management is a socio-technical phenomenon.&rdquo; Why do you think so?&nbsp; As technologists, we don't like to talk to people. That's the problem. If you look at the articles that people wrote back in the mid-90&rsquo;s about the data problems they were having, you'll realize that these are the same problems that I heard yesterday. So we&rsquo;ve had 30 years of dealing with the same problems. Why?&nbsp; I would argue that with all the technology pushing that we have done up to now, we've hit a barrier. If we continue to think that we just lack the right technology, we're literally going to drive ourselves insane. We need to have a paradigm shift, and that paradigm shift is to start inputting more of the social aspects into data management. If you think about it, we as technologists, define success from a technological perspective. But what about social success? At the end of the day, organizations need to answer critical business questions in order to make money and save money.&nbsp; Technology is a means to an end. I think we have focused a lot on technology, but haven&rsquo;t really zoomed out to understand what those ends are, which is the social aspect. We really need to understand the people, the process, and all the context around that stuff. I would argue that the successful technologists are the ones who become more social. What would be your advice to data practitioners and data owners so they will be able to adapt to this socio-technical approach to data? First of all, we talk about data literacy all the time. I think we should start talking about business literacy. So that means not just the data people, but everybody in the company needs to understand how the business works. So, for me, the number one thing is talking about business literacy.&nbsp; Number two: Get comfortable outside of your comfort zone. If you live in your bubble and feel great, but you want to be successful, you have to get out of that bubble and feel uncomfortable and learn from that discomfort. The third thing is to understand the problem holistically. Keep asking &ldquo;why?&rdquo; Successful data practitioners are the ones who are a lot like therapists. They ask a lot of questions. You have to do the same with folks around you. When someone asks you to pull certain data, ask, &ldquo;Why do you need this data?&rdquo; The most common answer you&rsquo;re going to get is to democratize data, so that they can be more data driven. Of course, we need to be all of that. But what&rsquo;s unique here? At some point you may realize, oh, that person really doesn't know. That's why it's important for everyone to understand how the business works.&nbsp; Next is knowledge. We live in a world of just data. We need to move to a world of knowledge and understanding of the people, context, and relationships surrounding data. There should be a balance between efficiency and resilience. We are all so focused on having short-term solutions that are very efficient, but by doing that, we are not setting the foundations for being resilient. Think about when the pandemic started - we ran out of toilet paper. Why? Because we had an efficient supply chain, but not a resilient supply chain. The Suez canal is super efficient to go through, but when a boat changes its angle and gets stuck, the entire economy of the world shuts down! Hence, the supply chain was not resilient.&nbsp; How has the role of data scientists changed from when you first started studying computer science back in 2004 to now? And what's your take on the role of Data scientist and Data engineer? Let&rsquo;s go back in time and look at it decade by decade. The web was invented by Tim Berners-Lee in early 1990's, and it became very popular specially with the rise of e-commerce. This is when we started generating data for the web, which took us into the big data era. Now you have a lot of data, and in order to be able to manage these clusters, the role of data engineer was established. Then in 2010&rsquo;s, the Cloud became popular along with AI/ML and deep learning. This is when we started getting insights from all the data that we had harvested. With this, machine learning also becomes a commodity. Anybody can become a data scientist, because all they have to do is learn various frameworks. You don't really go and develop these models. During the 2010&rsquo;s, we heard a lot about the 80/20 principle. Spending 80% of your time cleaning the data and 20% of your time doing analysis. We went from ETL to ELT - dump the data without understanding what it means, and then you're trying to make sense of that data. We till haven't done that knowledge modeling and semantic work. So then we start seeing data scientists complaining about this, and that's what happened from 2010 to up to now. The shift now is that people are starting to realize that they need to understand if they can rely on this data or not, and what the data actually means. The folks at dbt have really pushed the idea of transformations. With the rise of analytics engineers, we're starting to understand more about the meaning of data. The data engineer of the 2000s (Hadoop times) is different from the data engineer of today. It's different because we are no longer using Hadoop, but the principles are the same. From the principles perspective, we need to understand the inputs and the outputs, and forget about the name of the technology. In ETL, the E and the L are extracting &amp; loading. The principle is that you're moving data. You need to have very strong people who are making sure that the data is moving reliably with the use of technologies like data observability and monitoring, etc.&nbsp; The second thing is to make sense of that data, that&rsquo;s where data transformations and modelling come into play. Analytics engineers are responsible for this task, and I like to call them knowledge scientists. So if we look at history, we focused on the start and end of the data. Where does this data come from and where does this data go. This left us with a big gap in terms of understanding the data. And now we have finally realized that we have this gap. You have built data products for your entire career. So what are your thoughts on creating an amazing user experience while building a data product? When you say the word &lsquo;amazing,&rsquo; for me, it is my experience buying anything on Amazon. I can go and click a couple of buttons and get anything shipped anywhere in the world. 99% of the time, when I buy something on Amazon, I'm happy. We could not do this five-ten years ago. So why can't we have that amazing experience with data within our organizations?&nbsp; Now, Amazon knows me as the end consumer. As an end consumer, I don't care about the data pipeline. We as data people live in this bubble of data, where people think they're the greatest thing. Within our bubble we are, but lets be honest, outside of it, we probably aren&rsquo;t.The majority of people don't care about the work we do because it&rsquo;s a means to an end. So that's the first thing - we need to get off our high horse and stop thinking that we are the greatest thing in the world and learn about the business.&nbsp; Secondly, we need to understand who our consumers are. We need to provide an amazing platform to search for products. That's what a data catalog is. I can have the best data catalog, but if I don't have products that can be sold or found, the catalog doesn't work. So how do I get those data products? Organizations must understand business user&rsquo;s problems. What are their needs? Who are the types of users they have? Doing market research and understanding your consumers is incredibly important. Tim Gasper, a colleague of mine at data.world, and I, we do a podcast together called Catalog and Cocktails, and we came up with this framework called the Data Product ABCs. The ABCs are: Accountability, Boundaries, Contracts and Expectations, Downstream Consumers, and Explicit Knowledge. <ol> <li aria-level="1">Accountability is: Who is responsible for this? Who gets the requirements? When it breaks, who's on call to fix it?&nbsp;</li> <li aria-level="1">Boundaries include: What is this thing? For whom it is made for, and for whom is it not?</li> <li aria-level="1">Contracts and expectations include the expected use of that product. For every product, there is a warranty saying it can be used only for xyz things. The warranty would not be used if, say, I throw it against the wall. For data, what are these policies? What are the securities? What is its quality? We all love to talk about quality, but quality depends on the different users and on different use cases.&nbsp;</li> <li aria-level="1">Downstream Consumers means understanding the users and programs that depend on the Data Product. Who are the current consumers of the Data Product? Who are the potential consumers? What are the use cases that have been considered?</li> <li aria-level="1">Explicit Knowledge is making the meaning and semantics of the Data Product very clear: What is the meaning of the Data Product? What is the schema/ontology with corresponding definitions? What are the data constraints?</li> </ol> So that's the experience that we should be having with data at the end of the day. The data catalog is like that shopping experience. There are two lenses of a catalog. The lens that I just described is a consumer lens. They don&rsquo;t need to go into the details of the data quality. All they want to know is if I can go use this for my use case. The second lens of the catalog is for the technical folks. They are going to get all this data from different places and are going to have to deal with bad data. They really need to be able to understand where the data comes from, because they're the ones who are trying to make sense of what this data means.&nbsp; Recently, there has been an explosion of tools in MDS. What are the areas that you think are not very well served by the whole modern data stack? My honest answer is everything's there. There's already too much, and it&rsquo;s great. What we need right now is to focus on the social aspect of things, and for that, we don&rsquo;t need any more technology or any more stack. We're seeing that every little feature is almost becoming its own category and its own set of companies. That's too much. We just have enough of the technology and I would say we don't need more. RAPID FIRE What's your favorite tool in the whole modern data stack? Snowflake&nbsp; Any recommendations on newsletters or podcasts or any good resources to brush up data knowledge? I obviously need to say Catalog and Cocktails, our honest, no-bs, non-salesy data podcast. My main source is LinkedIn. I just follow a bunch of people on LinkedIn who have amazing ideas to share, and then the LinkedIn algorithm recommends a bunch of people that I can follow.I recommend following all the guests that we&rsquo;ve had on our podcast.&nbsp; Finally, I always recommend reading the book Software Wasteland by Dave McComb. This is a must read book by every single data professional in the world because it describes the root of data problems: an application-centric architecture. One thing you love and hate about your job I am extremely lucky to be exposed to so many problems, and this is what excites me. I get to talk to so many customers and understand what people's problems are. The thing that I dislike - well, let me be cliche: Love what you do and do what you love. You&rsquo;ll never work a day in your life, and hopefully you&rsquo;ll get paid for it.&nbsp;

Meet Juan: who turned his Ph.D. research into a successful company, Capsenta, and now is a Principal Data Scientist at data.world. He believes data management needs a paradigm shift and that shift is including social aspect to the business.

Amazing People in Data

The Socio-Technical approach to data

Building Modern Data Stack from scratch with Vladyslav

On being a full stack data scientist

Unbundling streaming stack with Dunith

Unsolved Problems of Modern Data Stack with Kelly